*Post by David Brown**Post by DFS*https://www.calculatorsoup.com/calculators/statistics/descriptivestatistics.php

My code compiles and works fine - every stat matches - except for one

anomaly: when using a dataset of consecutive numbers 1 to N, all

values > 40 are flagged as outliers. Up to 40, no problem. Random

numbers dataset of any size: no problem.

And values 41+ definitely don't meet the conditions for outliers

(using the IQR * 1.5 rule).

Very strange.

before: char outliers[100];

after : char outliers[100] = "";

And the problem went away. Reset it to before and problem came back.

Makes no sense. What could cause the program to go FUBAR at data

point 41+ only when the dataset is consecutive numbers?

Also, why doesn't gcc just do you a solid and initialize to "" for you?

It is /really/ difficult to know exactly what your problem is without

seeing your C code! There may be other problems that you haven't seen yet.

The outlier section starts on line 169

=====================================================================================

//this code is hereby released to the public domain

#include <stdlib.h>

#include <stdio.h>

#include <math.h>

#include <string.h>

#include <time.h>

/*

this program computes the descriptive statistics of a randomly

generated set of N integers

1.0 release Dec 2020

2.0 release Jun 2024

used the population skewness and Kurtosis formulas from:

https://www.calculatorsoup.com/calculators/statistics/descriptivestatistics.php

also test the results of this code against that site

compile: gcc -Wall prog.c -o prog -lm

usage : ./prog N -option (where N is 2 or higher, and option is -r or

-c or -o)

-r generates N random numbers

-c generates consecutive numbers 1 to N

-o generates random numbers with outliers

*/

//random ints

int randNbr(int low, int high) {

return (low + rand() / (RAND_MAX / (high - low + 1) + 1));

}

//comparator function used with qsort

int compareint (const void * a, const void * b)

{

if (*(int*)a > *(int*)b) return 1;

else if (*(int*)a < *(int*)b) return -1;

else return 0;

}

int main(int argc, char *argv[])

{

if(argc < 3) {

printf("Missing argument:\n");

printf(" * enter a number greater than 2\n");

printf(" * enter an option -r -c or -o\n");

exit(0);

}

//vars

int i=0, lastmode=0;

int N = atoi(argv[1]);

int nums[N];

double sumN=0.0, median=0.0, Q1=0.0, Q2=0.0, Q3=0.0, IQR=0.0;

double stddev = 0.0, kurtosis = 0.0;

double sqrdiffmean = 0.0, cubediffmean = 0.0, quaddiffmean = 0.0;

double meanabsdev = 0.0, rootmeansqr = 0.0;

char mode[100], tmp[12];

//generate random dataset

if(strcmp(argv[2],"-r") == 0) {

srand(time(NULL));

for(i=0;i<N;i++) { nums[i] = randNbr(1,N*3); }

printf("%d Randoms:\n", N);

printf("No commas : "); for(i=0;i<N;i++) { printf("%d ", nums[i]); }

printf("\nWith commas: "); for(i=0;i<N;i++) { printf("%d,", nums[i]); }

qsort(nums,N,sizeof(int),compareint);

printf("\nSorted : "); for(i=0;i<N;i++) { printf("%d ", nums[i]); }

printf("\nSorted : "); for(i=0;i<N;i++) { printf("%d,", nums[i]); }

}

//generate random dataset with outliers

if(strcmp(argv[2],"-o") == 0) {

srand(time(NULL));

nums[0] = 1; nums[1] = 3;

for(i=2;i<N-2;i++) { nums[i] = randNbr(100,N*30); }

nums[N-2] = 1000; nums[N-1] = 2000;

printf("%d Randoms with outliers:\n", N);

printf("No commas : "); for(i=0;i<N;i++) { printf("%d ", nums[i]); }

printf("\nWith commas: "); for(i=0;i<N;i++) { printf("%d,", nums[i]); }

qsort(nums,N,sizeof(int),compareint);

printf("\nSorted : "); for(i=0;i<N;i++) { printf("%d ", nums[i]); }

printf("\nSorted : "); for(i=0;i<N;i++) { printf("%d,", nums[i]); }

}

//generate consecutive numbers 1 to N

if(strcmp(argv[2],"-c") == 0) {

for(i=0;i<N;i++) { nums[i] = i + 1; }

printf("%d Consecutive:\n", N);

printf("No commas : "); for(i=0;i<N;i++) { printf("%d ", nums[i]); }

printf("\nWith commas : "); for(i=0;i<N;i++) { printf("%d,", nums[i]); }

}

//various

for(i=0;i<N;i++) {sumN += nums[i];}

double min = nums[0], max = nums[N-1];

//calc descriptive stats

double mean = sumN / (double)N;

int ucnt = 1, umaxcnt=1;

for(i = 0; i < N; i++)

{

sqrdiffmean += pow(nums[i] - mean, 2); // for variance and sum squares

cubediffmean += pow(nums[i] - mean, 3); // for skewness

quaddiffmean += pow(nums[i] - mean, 4); // for Kurtosis

meanabsdev += fabs((nums[i] - mean)); // for mean absolute deviation

rootmeansqr += nums[i] * nums[i]; // for root mean square

//mode

if(ucnt == umaxcnt && lastmode != nums[i])

{

sprintf(tmp,"%d ",nums[i]);

strcat(mode,tmp);

}

if(nums[i]-nums[i+1]!=0) {ucnt=1;} else {ucnt++;}

if(ucnt>umaxcnt)

{

umaxcnt=ucnt;

memset(mode, '\0', sizeof(mode));

sprintf(tmp, "%d ", nums[i]);

strcat(mode, tmp);

lastmode = nums[i];

}

}

// median and quartiles

// quartiles divide sorted dataset into four sections

// Q1 = median of values less than Q2

// Q2 = median of the data set

// Q3 = median of values greater than Q2

if(N % 2 == 0) {

Q2 = median = (nums[(N/2)-1] + nums[N/2]) / 2.0;

i = N/2;

if(i % 2 == 0) {

Q1 = (nums[(i/2)-1] + nums[i/2]) / 2.0;

Q3 = (nums[i + ((i-1)/2)] + nums[i+(i/2)]) / 2.0;

}

if(i % 2 != 0) {

Q1 = nums[(i-1)/2];

Q3 = nums[i + ((i-1)/2)];

}

}

if(N % 2 != 0) {

Q2 = median = nums[(N-1)/2];

i = (N-1)/2;

if(i % 2 == 0) {

Q1 = (nums[(i/2)-1] + nums[i/2]) / 2.0;

Q3 = (nums[i + (i/2)] + nums[i + (i/2) + 1]) / 2.0;

}

if(i % 2 != 0) {

Q1 = nums[(i-1)/2];

Q3 = nums[i + ((i+1)/2)];

}

}

// outliers: below Q1−1.5xIQR, or above Q3+1.5xIQR

IQR = Q3 - Q1;

char outliers[200]="", temp[10]="";

if (N > 3) {

//range for outliers

double lo = Q1 - (1.5 * IQR);

double hi = Q3 + (1.5 * IQR);

//no outliers

if ( min > lo && max < hi) {

strcat(outliers,"none (using IQR * 1.5 rule)");

}

//at least one outlier

if ( min < lo || max > hi) {

for(i = 0; i < N; i++) {

double val = (double)nums[i];

if(val < lo || val > hi) {

sprintf(temp,"%.0f ",val);

temp[strlen(temp)] = '\0';

strcat(outliers,temp);

}

}

strcat(outliers," (using IQR * 1.5 rule)");

}

outliers[strlen(outliers)] = '\0';

}

stddev = sqrt(sqrdiffmean/N);

kurtosis = quaddiffmean / (N * pow(sqrt(sqrdiffmean/N),4));

//output

printf("\n--------------------------------------------------------------\n");

printf("Minimum = %.0f\n", min);

printf("Maximum = %.0f\n", max);

printf("Range = %.0f\n", max - min);

printf("Size N = %d\n" , N);

printf("Sum N = %.0f\n", sumN);

printf("Mean μ = %.2f\n", mean);

printf("Median = %.1f\n", median);

if(umaxcnt > 1) {

printf("Mode(s) = %s (%d occurrences ea)\n", mode,umaxcnt);}

if(umaxcnt < 2) {

printf("Mode(s) = na (no repeating values)\n");}

printf("Std Dev σ = %.4f\n", stddev);

printf("Variance σ^2 = %.4f\n", sqrdiffmean/N);

printf("Mid Range = %.1f\n", (max + min)/2);

printf("Quartiles");

if(N > 3) {printf(" Q1 = %.1f\n", Q1);}

if(N < 4) {printf(" Q1 = na\n");}

printf(" Q2 = %.1f (median)\n", Q2);

if(N > 3) {printf(" Q3 = %.1f\n", Q3);}

if(N < 4) {printf(" Q3 = na\n");}

printf("IQR = %.1f (interquartile range)\n", IQR);

if(N > 3) {printf("Outliers = %s\n", outliers);}

if(N < 4) {printf("Outliers = na\n");}

printf("Sum Squares SS = %.2f\n", sqrdiffmean);

printf("MAD = %.4f (mean absolute deviation)\n",

meanabsdev / N);

printf("Root Mean Sqr = %.4f\n", sqrt(rootmeansqr / N));

printf("Std Error Mean = %.4f\n", stddev / sqrt(N));

printf("Skewness γ1 = %.4f\n", cubediffmean / (N *

pow(sqrt(sqrdiffmean/N),3)));

printf("Kurtosis β2 = %.4f\n", kurtosis);

printf("Kurtosis Excess α4 = %.4f\n", kurtosis - 3);

printf("CV = %.6f (coefficient of variation\n",

sqrt(sqrdiffmean/N) / mean);

printf("RSD = %.4f%% (relative std deviation)\n", 100 *

(sqrt(sqrdiffmean/N) / mean));

printf("--------------------------------------------------------------\n");

printf("Check results against\n");

printf("https://www.calculatorsoup.com/calculators/statistics/descriptivestatistics.php");

printf("\n\n");

return(0);

}

=====================================================================================

*Post by David Brown*Non-static local variables without initialisers have "indeterminate"

value if there is no initialiser. Trying to use these "indeterminate"

values is undefined behaviour - you have absolutely no control over what

might happen. Any particular behaviour you see is done to luck from the

rest of the code and what happened to be in memory at the time.

In 2024 that's surprising. I can't be the only one to forget to

initialize a char[] variable.

*Post by David Brown*There is no automatic initialisation of non-static local variables,

because that would often be inefficient.

It would've saved me half an hour of frustration.

Now I'm getting 'stack smashing detected' errors (after the program runs

correctly) when using datasets of consecutive numbers.

hmmmm 2 issues in a row using consecutives - that's a clue!

*Post by David Brown*The best way to avoid errors

like yours, IMHO, is not to declare such variables until you have data

to put in them - thus you always have a sensible initialiser of real

data. Occasionally that is not practical, but it works in most cases.

Data is definitely going in them: either the value 'none' or a list of

the outliers and some text.

*Post by David Brown*For a data array, zero initialisation is common. Typically you do this

int xs[100] = { 0 };

That puts the explicit 0 in the first element of xs, and then the rest

of the array is cleared with zeros.

I recommend never using "char" as a type unless you really mean a > character, limited to 7-bit ASCII. So if your "outliers" array really

is an array of such characters, "char" is fine. If it is intended to be

numbers and for some reason you specifically want 8-bit values, use

"uint8_t" or "int8_t", and initialise with { 0 }.

I did mean characters, limited to: 0-9a-zA-Z()

I think I'm using the char variable correctly.

sprintf(tempchar,"%d ",outlier);

strcat(char,tempchar);

*Post by David Brown*A major lesson here is to learn how to use your tools. C is not a

forgiving language. Make use of all the help your tools can give you -

enable warnings here. "gcc -Wall" enables a range of common warnings

with few false positives in normal well-written code, including ones

that check for attempts to read uninitialised data.

I always use -Wall, and I was using it here.

"-Wextra" enables a

*Post by David Brown*slew of extra warnings. Some of these will annoy people and trigger on

code they find reasonable, while most are good choices for a lot of code

- but personal preference varies significantly. And remember to enable

optimisation, since it makes the static checking more powerful.

Just did this:

gcc -Wall -Wextra -O3 mmv2.c -o mmv2 -lm

and no warnings or errors at all.

But: it now aborts near the front when using consecutive data points

(but not randoms).

*** buffer overflow detected ***: terminated

Aborted

I'm actually happy about that. I should be able to find and fix it.

*Post by David Brown*If you /really/ want gcc to zero out such local data automatically, use

"-ftrivial-auto-var-init=zero". But it is much better to use warnings

and write correct code - options like that one are an addition to

well-checked code for paranoid software in security-critical contexts.

Great answer! I can always count on D Brown for excellent advice.

Thank you.