Thread: For Loop for an m-by-1 matrix

Hello guys, I can not figure out how to make a for loop for a m-by-1 matrix (hence a 1 column vector) with which I start at the second element and subtract the previous element and then repeat this operation on the following elements.
How can I do this ?

for example I have an excel column called "Ref" that is made by these numbers:

Ref
839825
851892
851955
851969
851964
851518
851953
850544

I need something that in matlab will do for me :

851892-839825 =
851955-851892 =

ecc...
and create another column called :

Ref(j) - Ref(j-1)wih under it all the calcutated subtractions

Any help please ?
Thnak you

You can do this with a loop, or with vectorized code (better option) - see both examples below.

Code:

clc
Ref = [839825 851892 851955 851969 851964 851518 851953 850544]';

%With  loop
A = zeros(numel(Ref)-1,1);
for i = 1:numel(A)
    A(i) = Ref(i+1)-Ref(i)
end

%the Matlab way
B = Ref(2:end)-Ref(1:end-1)

Elbarto

Hi Elbarto, thank you so much.
I would like to learn what you did.
Can u pls explain me what all those commands mean and say to matlab ?
Can u suggest me a book to buy or electronic, where I could learn these matlab language codes ?

I have to do other steps after this step, and I am sure I do not have enough knowledge to do them.
I mean, I found all the basic matlab things, but when I want something more specific, I can not find it. Today I spent 4 hour in trying to find the correct matlab language codes to do what you did for me in 1 second.

Can u pls give me some suggestions ?
Thank you
Marco

Originally Posted by marcolisi

Hi Elbarto, thank you so much.
I would like to learn what you did.
Can u pls explain me what all those commands mean and say to matlab ?
Can u suggest me a book to buy or electronic, where I could learn these matlab language codes ?

I have to do other steps after this step, and I am sure I do not have enough knowledge to do them.
I mean, I found all the basic matlab things, but when I want something more specific, I can not find it. Today I spent 4 hour in trying to find the correct matlab language codes to do what you did for me in 1 second.

Can u pls give me some suggestions ?
Thank you
Marco

If you Google for "introduction to matlab" you will find many free books that explain what you need.

CB

No problem - starting off with any programming language is a challenge if you haven't got any prior experience so I will try give you a quick run down on what is happening in this example with matlab. I have added a few comments to my code - let me know what you cant understand and I will try help you out.

As CB said, there is plenty on the net and that is where I learnt everything I know about matlab although when I first started off I was in the same boat as you and found some great advice on the forrums. If you have any trouble, keep setting out good posts like you have done above (ps try use code tags when posting code - its the # button) and you should get some helpful advice from the members of this forum.

Good luck

Regards Elbarto

Code:

clc%clear the command line so there is no old output there

%numbers inside [] separated by a space or comma are all treated as
%elements of a row vector. if you separate them with a ; then they
%become elements of a column vector. Here I create a row vector
%and then use the ' at the end to transpose the row vector into a 
%column vector. Its just a shortcut for creating a column vector
%but you can create it by separating the items with ; if you like;
Ref = [839825 851892 851955 851969 851964 851518 851953 850544]';
%note the last ; at the end of the line means that the output wont
%be printed at the command line. take it off and see what happens.

%With  loop
%Here I first create a matrix with 1 column (so really it is a 
%column vector) and I now that there are going to be 1 less rows then
%the vector "Ref" so I use the function "numel" to count how many elements
%are in the "Ref" vector and subtract 1. So what I have done here is
%create a matrix of zeros in 7 rows and 1 column. This step isn't
%really crucial but it is good practice as it means that you have 
%preallocate a block of memory the correct size so you dont have to 
%keep resizing you matrix every time you want to add another row (do worry
%too much about this). You can also use length() or size() to find out 
%how big a matrix is but they all work in a different way so do some
%searching through the help files.
A = zeros(numel(Ref)-1,1);
%Now I create a vector of [1,2,3,4,5,6,7] and loop through each number
for i = 1:numel(A)
    %so row 1 in A will be the (2nd row of Ref - 1st row of Ref)
    A(i) = Ref(i+1)-Ref(i)
    %note this is the same as "A(i) = Ref(i+1,1)-Ref(i,1)" but I dont 
    %really need to put the 1 in the column possition because matlab
    %knows that A is a vector and not a matrix.
    
    %By the way, when you reference an element of a matrix in matlab it
    %is Matrix(row_number,col_number).
end

%now for the easy way (the Matlab way)
%"end" is a special word in matlab and if you use it when referencing
%a vector or matrix it will go the the end of the row or column. So 
%all I do below is say "take everything from the second row down, then
%subtract the numbers from the 1st to the second last row". I am just
%taking 2 slices of the original vector and getting matlab to do a 
%simple vector subtraction.
B = Ref(2:end)-Ref(1:end-1)

%Always try and use the second method which matlab calls "vectorized code".
%Matlab can perform the calculation a lot faster then if you do it using
%the first method although for trivial examples like this you wont notice
%any difference. If you had a few hundred thousand rows of data though,
%you will soon see the difference.

Hi, thank you s much for the explanations. The "vectorized code" is much easier for me to understand.

I think I have to do a step back.
WhatI mean is that everything start in my case from an excell spreadsheet (see attached file).
I import in matlab the excell file and in paticoular "sheet1" and I select "create vectors from each..." option.
When I import it, I get this matlab code :

Code:

function importfile(fileToRead1)
%IMPORTFILE(FILETOREAD1)
% Imports data from the specified file
% FILETOREAD1: file to read

% Auto-generated by MATLAB on 08-May-2010 17:17:39

% Import the file
sheetName='Sheet1';
[numbers, strings, raw] = xlsread(fileToRead1, sheetName);
if ~isempty(numbers)
newData1.data = numbers;
end

if ~isempty(strings) && ~isempty(numbers)
[strRows, strCols] = size(strings);
[numRows, numCols] = size(numbers);
likelyRow = size(raw,1) - numRows;
% Break the data up into a new structure with one field per column.
if strCols == numCols && likelyRow > 0 && strRows >= likelyRow
newData1.colheaders = strings(likelyRow, :);
end
end

% Create new variables in the base workspace from those fields.
for i = 1:size(newData1.colheaders, 2)
assignin('base', genvarname(newData1.colheaders{i}), newData1.data(:,i));
end

Than I am going to av in matlab workspace these 3 names :

BC, REF and TIME

If you see the xcell file attached and the sheet2, you are going to have an idea of what I need to do.

So, now in matlab , I use your matlab code :

Code:

B = Ref(2:end)-Ref(1:end-1)

to calculate :

Ref(j) - Ref(j-1)

than I calculate the absolute value of each element (Abs Ref(j) - Ref (j-1)) of the vector B using this code below:

Code:

C = abs (B)

than I want to calciulate the 95% of the " Abs Ref(j)- Ref(j-1) "

using this code :

Code:

Y = prctile(C,0.95)

question: should I use 0.95 or 95 to calcuate the percentile of 95% ? In excel I use 0.95

than I hae to calculate "Bad BC Data Removed" , meaning that in excel I use an if statement :

=IF(F3>G$2,"",C3)

that says that if for example a selected element in "Abs Ref(j) - Ref (j-1)" is higher than (>) the 95% percentile (95% Abs Ref(j)- Ref(j-1)), I am going to have a "blank cell", while instead if it is not >, I am going to use its "BC" value instead.

The problem now is , how do I do this in matlab ?
Which is the code in matlab to indicate each element of the vector ?

Than the next step after this is to calculate an average of the values before the blank cells (1 before the blank cell and 1 after it). The problem here is that I have to implement a matab code that :

1) if there is just 1 blank cell, it does for me the average of the number before the blank cell and the one after it

2) if instead there are 2 blank cells, it does for me a weighed average,taking the number before the 2 white cells (the 2 blank cels are called P11 and P12 for example in excel) and the one after the 2 blank cells and do of them the weighed averge like this : (P10+P13*2)/3

I know it is a lot to do, but which are the chapters that I should focus on to be able to implement a matlab code that could do this for me ?

Thank you
Marco

Originally Posted by elbarto

No problem - starting off with any programming language is a challenge if you haven't got any prior experience so I will try give you a quick run down on what is happening in this example with matlab. I have added a few comments to my code - let me know what you cant understand and I will try help you out.

As CB said, there is plenty on the net and that is where I learnt everything I know about matlab although when I first started off I was in the same boat as you and found some great advice on the forrums. If you have any trouble, keep setting out good posts like you have done above (ps try use code tags when posting code - its the # button) and you should get some helpful advice from the members of this forum.

Good luck

Regards Elbarto

Code:

clc%clear the command line so there is no old output there
 
%numbers inside [] separated by a space or comma are all treated as
%elements of a row vector. if you separate them with a ; then they
%become elements of a column vector. Here I create a row vector
%and then use the ' at the end to transpose the row vector into a 
%column vector. Its just a shortcut for creating a column vector
%but you can create it by separating the items with ; if you like;
Ref = [839825 851892 851955 851969 851964 851518 851953 850544]';
%note the last ; at the end of the line means that the output wont
%be printed at the command line. take it off and see what happens.
 
%With  loop
%Here I first create a matrix with 1 column (so really it is a 
%column vector) and I now that there are going to be 1 less rows then
%the vector "Ref" so I use the function "numel" to count how many elements
%are in the "Ref" vector and subtract 1. So what I have done here is
%create a matrix of zeros in 7 rows and 1 column. This step isn't
%really crucial but it is good practice as it means that you have 
%preallocate a block of memory the correct size so you dont have to 
%keep resizing you matrix every time you want to add another row (do worry
%too much about this). You can also use length() or size() to find out 
%how big a matrix is but they all work in a different way so do some
%searching through the help files.
A = zeros(numel(Ref)-1,1);
%Now I create a vector of [1,2,3,4,5,6,7] and loop through each number
for i = 1:numel(A)
    %so row 1 in A will be the (2nd row of Ref - 1st row of Ref)
    A(i) = Ref(i+1)-Ref(i)
    %note this is the same as "A(i) = Ref(i+1,1)-Ref(i,1)" but I dont 
    %really need to put the 1 in the column possition because matlab
    %knows that A is a vector and not a matrix.
 
    %By the way, when you reference an element of a matrix in matlab it
    %is Matrix(row_number,col_number).
end
 
%now for the easy way (the Matlab way)
%"end" is a special word in matlab and if you use it when referencing
%a vector or matrix it will go the the end of the row or column. So 
%all I do below is say "take everything from the second row down, then
%subtract the numbers from the 1st to the second last row". I am just
%taking 2 slices of the original vector and getting matlab to do a 
%simple vector subtraction.
B = Ref(2:end)-Ref(1:end-1)
 
%Always try and use the second method which matlab calls "vectorized code".
%Matlab can perform the calculation a lot faster then if you do it using
%the first method although for trivial examples like this you wont notice
%any difference. If you had a few hundred thousand rows of data though,
%you will soon see the difference.

marcolisi,

I will have a look at your code after work tomorrow as there is fair bit there to get through and its past my bed time. You might like to look into how to remove rows from a vector for example to get rid of any "bad" data for performing calculations.

Regards Elbarto

Here is my attempt. Have a look at row E of your spreadsheet - im not real sure what you have done there and it looks like the last row is incorrect hence you 95th percentile might be wrong.

Code:

clc;clear;
fname = 'try.xls';
sheetName='Sheet1';
xls = xlsread(fname, sheetName);
Ref = xls(:,2);%get second row
BC = xls(:,3);%get thrid row

%calc abs of difference
ABS_Refj_Refi = abs(Ref(2:end)-Ref(1:end-1));
%95th percentile
p = prctile(ABS_Refj_Refi,95);
%find all BC vals where ABS_Refj_Refi<95th percentile
C = BC(ABS_Refj_Refi<p);
%delete first row of vector if you want (you dont use it in excel)
C(1) = [];
%print vector
disp(C)

Check it out and let us know how you get on.

Regards Elbarto

Hi Elbarto, thank you for the code.

I am not sure what the code is going to do (just remove the BC values that are <95% ?).
If this is the case, this is "not" what I need.

I know that logically when we have a huge set of the data and among them we have few sparse BC datapoint that look wrong (the BC values are <95% percentile), the best thing to do is to remove them.....

The problem here is that the person I should do this for do not want this. He wants that whenever we find a BC value <95% percentile, we do NOT delete that point, but we calculate how that BC value should be, interpolating.

If u see the new attached excel file, you can see 1 full dataset and what we tried to do in excel but it did not work easily because it was maybe impossible to calculate the "if" statement, for the different cases.

The cases are :

we can get all good BC values and than at one point:

1) one wrong BC value ( see the "Bad BC data removed" column in excel, look for the blank cell/cells. When I have the blank cell, means that that BC data point was wrong)

or

2) two consecutive wrong BC values (two consecutive blank cells)

or even

3) three consecutive wrong BC values (three consecutive blank cells)

What he would like to do, is this:

for the case 1)"one wrong BC value" : just take the BC number/value before the only blank cell and take also the one after the blank cell and average them, so that we get a new BC value that should represent the correct BC value to use for the blank cell.

for the case 2)"two consecutive wrong BC values" : do a weighed average (P10+P13*2)/3 , where P10 and P13 are the excel cells (BC values) respectively before and after the 2 consecutive blank cells.

for the case 3)"three consecutive wrong BC values" : do the same thing of case 2) using the weighed average but extended over the 3 blank cells.

Now, coming back to the "matlab code" that you did for me,

a) when I insert the first part of the code :

Code:

clc;clear;
fname = 'try.xls';
sheetName='Sheet1';
xls = xlsread(fname, sheetName);
Ref = xls(:,2);%get second row
BC = xls(:,3);%get thrid row

I get this message:

Code:

??? Error using ==> xlsread at 219
XLSREAD unable to open file try.xls.
File C:\Users\Lupo\Documents\MATLAB\try.xls not found.

I do not uderstand your code, I mean, can I insert a code line in matlab telling to look at a specific folder where the excel file is ?
I do not understand the different passages of your matlab code above.

Anyway, I worked around it, just importing with the wizard the excel file in matlab, and then, I execute the second part of your code :

Code:

%calc abs of difference
ABS_Refj_Refi = abs(Ref(2:end)-Ref(1:end-1));
%95th percentile
p = prctile(ABS_Refj_Refi,95);
%find all BC vals where ABS_Refj_Refi<95th percentile
C = BC(ABS_Refj_Refi<p);
%delete first row of vector if you want (you dont use it in excel)
C(1) = [];
%print vector
disp(C)

that calculates for me:

- the absolute values of the differences

- the 95th percentile


Then I do not really understand this part of the code :

Code:

%find all BC vals where ABS_Refj_Refi<95th percentile
C = BC(ABS_Refj_Refi<p);
%delete first row of vector if you want (you dont use it in excel)
C(1) = [];

I mean, it should find for me all the BC values <95th percentile, but when I click on "C", I do not understand what list of numbers is that.

This step should be more something like:

- whenever comparing "ABS_Refj_Refi" to the "95th percentile" ,we get that the "ABS_Refj_Refi" value is > (higher than) the "95th percentile" , put a blank cell in the "Bad BC data removed" column for that value


- whenever comparing "ABS_Refj_Refi" to the "95th percentile" ,we get that the "ABS_Refj_Refi" value is < (lower than) the "95th percentile" , use the corresponding BC value in the "Bad BC data removed" column for that value

Thank you for your help
Marco

Marco,

try.xls needs to be in the same directory as the code that you are executing or on the search path. It might sound complicated but it is a very simple concept that google will help you understand. The error you are getting is because matlab cant find the file.

The Variable "C" is the same as the last column in your first spreadsheet, but in this case any bad data was removed. Matlab does really do "blank cells" in a matrix or vector (you can use a cell array but I avoid them if I can). Interpolating instead of deleting bad data is possible, and probably best done with a for loop as you suggested. This is a bit more involved tho as you will need to

1) Loop through data and find bad point
2)determine number of consecutive bad points
3)interpolate between points (see interp function)

It is certainly achievable. I started having a play around to see how you can find our how many times consecutive numbers are repeated (Captain Black could prob do it in one line of code). I used a recursive function to find the matrix, you will need to save the function in the same dirrectory as the other files with the correct file name:


My Code:

Code:

clc;clear;
fname = 'try.xls';
sheetName='Sheet1';
xls = xlsread(fname, sheetName);
Ref = xls(:,2);%get second row
BC = xls(:,3);%get thrid row

%calc abs of difference
ABS_Refj_Refi = abs(Ref(2:end)-Ref(1:end-1));
%95th percentile
p = prctile(ABS_Refj_Refi,85);

bad_flag = find(ABS_Refj_Refi > p)%position of bad data

bad_index_list = getNextDoor(bad_flag)%see my custom function
%we get all indexes that are bad and how many consective indexes there are.

my function (save as getNextDoor.m)

Code:

function a = getNextDoor(vec,mat)
%returns index of first index and how many occurances follow
%[first_index      number_of_consective_numbers)
if nargin == 1;mat = [];end

if isempty(vec);
    a = mat;
    return
end%base case
if isempty(mat) || vec(1) ~= mat(end,1)+mat(end,2)+1;
    mat = [mat;[vec(1) 0]];
    vec(1) = [];
    a = getNextDoor(vec,mat);
else
    vec(1) = [];
    mat(end,2) = mat(end,2) + 1;
    a = getNextDoor(vec,mat);
end

I have run out of time tonight to try and get a solution so you can have a play with the code and see how that goes. Matlab and excel give different values for the 95th percentile so I have used the 85th percentile to create more bad data while testing the code.


Regards Elbarto

(Sorry I made this post in a bit of a rush so it is pretty sloppy)

Hi Elbarto, thank you so much. I run your code today and it works great.

Anyway, tomorrow I am going to look at your code piece by piece , because I want to understan how you did it. I want to learn. I m sure I am going to hav some problem in understanding weverythig, so pls be patience if I gt back to you with questions. Please do not leave me alone. I am still missing the final part of the code , that I hink is going to be even more difficoul, but again, before going forward, I wish to stop a secon and learn from your notes...



Thank you for your time

Marco

No Problem Mate - Sorry I cant explain why matlab and excel give different values for the 95th percentile (I rarely use any statistical features in matlab).

Now that you have a list of all the indices of all the bad data and how many times it is repeated you can start to think about interpolating between the points. Of course you are 100% correct when you say that you should first try and understand the code I have posted as there might be some issues there that I have overlooked.

The getNextDoor.m is a recursive function and generally I try and avoid recursive functions as they can usually be rewritten using a standard loop (which is more efficient as I understand it). I probably shouldn't have introduced the topic of recursion since you are only picking up programming but I find it a useful tool for certain problems like the one you presented where you can write a seemly complicated function in only a few lines of code.

Good luck with matlab.


Regards Elbarto

Hi Elbarto, I am completelly lost.

Sorry....this is too complicated for me right now to understand.

the few things that I read and tried to understand are :

1)

Code:

function a = getNextDoor(vec,mat)

a = is the output

getbextdoor = is the function name

vec = is the input 1

Mat = is the input 2

why you chose vec and mat ?

nargin = calculates the numbers of the inputs

but how do you know how many they are ?
why is ==1 ?

mat = [] what does it mean ?

isempty = determine whether the array is empty or not. If it is 0, it is non empty . If it is 1, is empty

Anyway, I really do not know where to start from to understand these last codes that you sent.
They do the job perfectly, but for me right now , is like arabic.
I tried to look at the "recursive function" in matlab, but I can not figure out the meaning of each step of your code.

Can u pls give me some help when u have time and guide me through the code ? or do u have any advice ?

For the interpolation/loop, I have an idea of doing this :

Code:

consecutive_bad_BC: is the number of consecutive BC values whose ABS_refj_refi is greater than p

j: is the identifier of the last good BC

then it has to be true that the next good BC has an identifier of j + consecutive_bad_BC + 1

for k = 1 to consecutive_bad_BC

BC(J + k) = ((consecutive_bad_BC + 1 - k)*BC(j) + (k)*BC(j + consecutive_bad_BC +1))/(consecutive_bad_BC + 1)

next k 

k is a number from 1 to infinite that tells us at which repetition/iteration we are.


example:



ID          [BC]          condecutive_bad_BC         j      interpolation

1           1000     

2                                 1                  1

3            800

4            600

5                                 2                  4

6

7            300

8                                 3                  7

9

10

11           700



if I use this formula we can calculate/interpolate:

- the BC concentration (for 1 blank spot ID 2)


BC(J + k) = ((consecutive_bad_BC + 1 - k)*BC(j) + (k)*BC(j + consecutive_bad_BC +1))/(consecutive_bad_BC + 1)


and I substitute the values from the tables above, we have :

Concentration of BC for the "raw" number (1+1) = ((1+1-1) * black carbon concentration for the "raw" number 1 + 1* black carbon concentration for the "raw" number (1+1+1)) / (1+1)

hence

Concentration of BC for the "raw" number (2) = (1 * 1000 + 1 * black carbon concentration for the "raw" number 3) / 2

hence

Concentration of BC for the "raw" number (2) = (1000 + 1 * 800)/2

hence

Concentration of BC for the "raw" number (2) = (1000 + 800)/2 = 1800/2 = 900

where k = 1 because because we are at the first iteration/repetition



------------------------------------------

- now lets try the same formula to calculate/interpolate the next BC concentration (for the 2 blank spots ID 5 and 6)


Fort the first blank spot ID 5:

BC(J + k) = ((consecutive_bad_BC + 1 - k)*BC(j) + (k)*BC(j + consecutive_bad_BC +1))/(consecutive_bad_BC + 1)

Concentration of BC for the "raw" (4+1) = ((2+1-1) * black carbon concentration at raw number 4 + 1* black carbon concentration for the "raw" number (4+2+1))/ (2+1)



Concentration of BC at raw number 5 = (2*600 + 1*black carbon concentration for the "raw" number 7)/ 3


Concentration of BC at raw number 5 = (1200 + 1*300) /3 = (1200 + 300)/3 = 1500 / 3 = 500

we used k=1 because we are at the first iteration/repetition






For the second blsnk spot ID 6:

BC(J + k) = ((consecutive_bad_BC + 1 - k)*BC(j) + (k)*BC(j + consecutive_bad_BC +1))/(consecutive_bad_BC + 1)


Concentration of BC for the "raw" (4+2) = ((2+1-2) * black carbon concentration at raw number 4 + 2*black carbon concentration for the "raw" number (4+2+1))/ (2+1)


Concentration of BC at raw number 6 = (1*600 + 2*black carbon concentration for the "raw" number 7)/ 3


Concentration of BC at raw number 6 = (600 + 2*300) /3 = (600 + 600)/3 = 1200 / 3 = 400

we used k=2 because we are at the second iteration/repetition

but how do I pratically write this in matlab words ?

Please help me
I am willing to wait when you have time, so that maybe you could guide me through.

Thank you
Marco

"vec" and "mat" are just arbitrary names I gave. Basically when the user calls the function he only needs to provide one input. "nargin" is a key word inside a function which IS how many inputs have been provided, so if only 1 input is provided I create and empty matrix called "mat". It is important to initialize "mat" if its not provided otherwise matlab will give an error.

I will try rewrite the function non recursively when I get a chance as it is a really bad example for me to post and expect someone new to matlab to understand so sorry about that (take all the ; off the end of the lines in the function and run it and it might be a little clearer how it works).

Originally Posted by elbarto

"vec" and "mat" are just arbitrary names I gave. Basically when the user calls the function he only needs to provide one input. "nargin" is a key word inside a function which IS how many inputs have been provided, so if only 1 input is provided I create and empty matrix called "mat". It is important to initialize "mat" if its not provided otherwise matlab will give an error.

I will try rewrite the function non recursively when I get a chance as it is a really bad example for me to post and expect someone new to matlab to understand so sorry about that (take all the ; off the end of the lines in the function and run it and it might be a little clearer how it works).

Thank you so much. I appreciate all you do for me.
In the meantime, do you have any suggestion of what chapter I could read to be able to uderstand I can create the last part of the matlab code, based on the mathematical xample that I gave you of how to calcultate the normal average and the weighed average ?
Thank you