Anonymous:
I. SPECIFIC QUESITON. IDENTIFICATION RULES FOR SEM
This question is related to the SEM examples provided by the UCLA website at http://www.ats.ucla.edu/stat/sas/library/proc_calis.htm. In example 4, the path diagram shown involves 3 latent factors: F1 (mediator factor), F2 (endogenous factor) and F3 (exogenous factor). Looking at the SAS code, I noticed that the variance of the exogenous factor is not constrained to one, and instead, it is telling SAS to calculate the variance of F3 from the data (Std F3 = ef3). From my standpoint, this code is not following the rules of identification provided in class: 1) set variances of 1 for latent variables that are completely exogenous; and 2) allow correlations among exogenous variables. Am I right? So I ran the code on SAS to see if I had a non- identifiability issue (which I shouldn't have since the number of free-parameters is 3 (F3, D1 and D2 right?) and, this is not exceeding 3(3-1)/2 = 3, (where m=3). The code used was the following ( you may want to look at the attached file):
data power(TYPE=COV);
_type_ = 'cov';
input _name_ $ v1-v6;
datalines;
v1 11.834 . . . . .
v2 6.947 9.364 . . . .
v3 6.819 5.091 12.532 . . .
v4 4.783 5.028 7.495 9.986 . .
v5 -3.839 -3.889 -3.841 -3.625 9.610 .
v6 -2.189 -1.883 -2.175 -1.878 3.552 4.503
;
run;
proc calis cov data=power method = ml nobs = 932;
lineqs
V1 = F1 + E1,
V2 = .833 F1 + E2,
V3 = F2 + E3,
V4 = .833 F2 + E4,
V5 = F3 + E5,
V6 = a6 F3 + E6,
F1 = c1 F3 + D1,
F2 = c2 F1 + c3 F3 + D2;
std
D1 - D2 = ed:,
F3 = ef3, /* HERE is my concern */
E1 = ee1,
e3 = ee1,
e2 = ee2,
e4 = ee2,
e5 = ee3,
e6 = ee4;
cov
E1 E3 = theta1,
E2 E4 = theta1;
run;
When I looked at the SAS results, I first checked if I had any warning message in the Log, and I got the following:
WARNING: Shorter parameter list than variable list in STD statement. The parameter list is
filled up with 1 entries ed . .
NOTE: ABSGCONV convergence criterion satisfied.
I assumed that the Warning is because SAS didn't like the specification in the STD statement: " D1 - D2 = ed:," right? So I just changed the code to "D1 - D2 = ed1-ed2," and the message did not appear again. Additionally, looking at the Note, SAS tells me that the ABSGCONV convergence criterion is satisfied, so this means that the model is identifiable. However, this ABSGCONV criterion seems to be different from the GCONV convergence criterion that we saw in class. I am not sure if this difference in convergence criterion represents a problem. Should this be a concern? Now, referring to the fit statistics I got the following: RMR = 0.1507; Chi-Sq = 13.4764 DF=9 and RMSEA=0.0231. This tells me that the model fits quite well. Should I feel confident with this model even if the identification rules were not used?
Well, I tried to answer that question to myself by changing the code using the restriction Var(F3) =1. That is, under the STD statement, I only changed from "F3=ef3" to "FE=1". However, I expected to have better results, but I got this warning message "The central parameter matrix _PHI_ has probably 1 negative eigenvalue(s). " This definitely looks like something is wrong. But I am not sure what it is. After I got this message I looked at the Fit statistics and I did get worse results: RMR=1.3593, Chi-Squared= 178.2328, DF= 10, and RMSEA=0.1344.
Given the results, I can say that using the identification rules made the model fit worse! I am confused with these results. I do not understand what is going on here: for one side, the example 4 does not seem to follow the identification rules, and yet I got a good fit; and from the other, when I constrained the model, instead of getting better results, I got worse results. Is this because in this example there is only one exogenous latent variable? Under what conditions should I use the identification rules? I am assuming that whatever the reason is, it does not have to do with the fact that the code is using the covariance matrix as an input. Am I Right?
You said,
"From my standpoint, this code is not following the rules of identification provided in class: 1) set variances of 1 for latent variables that are completely exogenous; and 2) allow correlations among exogenous variables. Am I right?"
Right, they did (1) differently. The rules I set out are not set in stone - there are alternatives as you can see! The main rule is to ensure identifiability, and there are different ways to free up parameters.
Then you said, "So I ran the code on SAS to see if I had a non- identifiability issue (which I shouldn't have since the number of free-parameters is 3 (F3, D1 and D2 right?) and, this is not exceeding 3(3-1)/2 = 3, (where m=3)."
Ok be careful here. There are two issues, (1) # parms overall and (2) #parms in the model relating latent variables. Overall, there are 6*7/2=21 free parameters. They are estimating 11. Also, note that the condition *parms <= p(p+1)/2 is a *necessary condition* but not a *sufficient condition* for identifiability. Also, F3, D1 and D2 are not parameters, they are latent vriables. The parameters are the loadings, variances and covariances that are estimated by the model.
Here, they "violated" a major rule I gave in class, namly that the number of oarameters in the structural part should be less than or equal to m(m-1)/2, or <= 3 in this case. They are estimating 6 parameters in the structural part - c1, c2, c3, ed1, ed2, and ef3. This is ok because these parameters are constrained in the entire system - notice that the model is not a usual "CFA" form in the manifet part. There are many constraints. These constraints dictate constraints on the structural parameters as well so that the model is identifiable.
It is complex and difficult to identify exactly when the model is identifiable and not. This example is a little more advanced, and illustrates some of the finer points about identifiability.
Then you said, "I assumed that the Warning is because SAS didn't like the specification in the STD statement: " D1 - D2 = ed:," right?"
It's not a problem - they just named the variances ed1 and ed2. See the output.
Then you said "SAS tells me that the ABSGCONV convergence criterion is satisfied, so this means that the model is identifiable. However, this ABSGCONV criterion seems to be different from the GCONV convergence criterion that we saw in class."
Actually, this message does not ensure identifiability. You will also get the same message in non-identifiable situations. It's the line about "linear dependencies" that indicates non-identifiability. Notice that if you allow the usual CFA model form, where you replace their code
V1 = F1 + E1,
V2 = .833 F1 + E2,
with what we learned
V1 = b1 F1 + E1,
V2 = b2 F1 + E2,
then you get the error message that indicates non-identifiability:
NOTE: Covariance matrix for the estimates is not full rank.
NOTE: The variance of some parameter estimates is zero or some parameter estimates are linearly
related to other parameter estimates as shown in the following equations:
ed1 = 4.182315 + 0.125729 * b1 + 0.107435 * b2
+ 0.075472 * c2 + 0.0628
Then you said "ABSGCONV criterion seems to be different from the GCONV convergence criterion that we saw in class. I am not sure if this difference in convergence criterion represents a problem. Should this be a concern? "
No, SAS has different defaults depending on the problem. This might be related to the use of covariance matrix rather than correlation matrix. No concern. It's good that it converged, by whatever defult SAS selects.
Then you said, "Now, referring to the fit statistics I got the following: RMR = 0.1507; Chi-Sq = 13.4764 DF=9 and RMSEA=0.0231. This tells me that the model fits quite well. Should I feel confident with this model even if the identification rules were not used?"
Sure, the model is fine. My "rules" again are not hard and fast. There are many ways to re-parameterize that give equivalent results.
Side Note: RMR seems "large," but refers to the covariance matrix rather than the correlaiton matrix, so there is no standard for RMR in the case (ie, .05 is not relevant as a threshold here, since .1507 refers to differences between covariances, rather than differences between correlations.)
Then you said, "Well, I tried to answer that question to myself by changing the code using the restriction Var(F3) =1. That is, under the STD statement, I only changed from "F3=ef3" to "FE=1". However, I expected to have better results,"
Why do expect better? When you constrain, the fit always gets worse! It's just like regression - Rsquare always decreases when you delete an X variable (or, in SEM terms, when you constrain its beta to be equal to 0). Yes, this is an interesting example! Again, it's a little more advanced, illustrating some finer points in identification. It might be best to stick to the simpler, more standard examples. Again, the resolution is in the constraints dictated by the whole system of equations. The manifest part has various constraints that constrain the parameters in the structural part. It's clever example, suitable for advanced learning, but does cause confusion at the start of learning about this material.
100 100 100 100
II. GENERAL QUESTION. RMSEA Issues
My questions are related to the article "An Empirical Evaluation of the Use of Fixed Cutoff Points in RMSEA Test Statistic in Structural Equation Models".
1) The RMSEA formula expressed in the article involves the term N-1, whereas the formula given in class involves n. I am assuming that N and n stands for the sample size of the dataset. As far as I understand, the distinction between using N-1 and n depends on the type of covariance matrix we are using. That is, we use N-1 if the COV or CORR matrix is analyzed, and we use only n if the uncorrected correlation or covariance matrix. Is this correct? However, I do not exactly see the difference between an uncorrected and a corrected correlation or covariance matrix. Does it have to do with the standardized issue?
2) According to my understanding, the RMSEA index is a standardized version of the Chi-Squared index. We learned that the Chi-Squared gets larger as the sample size increases and this is not a desirable behavior of a fit index, right? Thus, we need fit indices that converge as the sample size increases. I thought that the RMSEA solved this problem of the Chi-squared by getting rid of the "n" in the RMSEA formula. And by that, I automatically implied that the RMSEA index is not being sensible to the sample size. After reading the paper, I am not sure about that. In the article, it is mentioned several times that the RMSEA tends to over-reject with small sample sizes. So that means that it is sensible to the sample sizes, right? Well, as a matter of fact the authors mention that the RMSEA is dependent on the sample size, model misspecification and degrees of freedom. So it seems that the RMSEA solves the problem of convergence, but it seems that a good interpretation of the RSMEA is subject to the experience of the researcher. So I wonder if whenever I have small sample sizes should I focus more on the Chi-Square index, and with larger sample sizes should I focus more on the RMSEA index? And then, how large should n need to be in order to be considered large?
3) The authors also mentioned the conclusions of a study carried out by Nevit and Hancock (2000). These conclusions are related to the difficulty of evaluating a hypothesis test for misspecified models under nonnormal conditions. If I am understanding this correctly, then the hypothesis Ho: S = S(q) may not be true if the manifest variables do not have a normal behavior and/or the estimated parameters are incorrect. Is this right? Then, it is not clear to me why the normality condition of the manifest variables is involved if the model being tested involves latent variables? Isn't the structure of S implied by the SEM model we are specifying? Should we check the normality assumption before any attempt of SEM analysis?
4) My last question is concerned with the method of analysis. The authors used 84 experimental conditions and 500 replications for each condition. Then they did hypothesis testing. Well, we learned the problem implied with multiple hypothesis testing: the probability of committing at least one type I error increases. Thus, the more tests I perform the more probability of making a false conclusion. False rejections are easily observed on the graphs shown in the article. Shouldn't this issue be considered? In order to do that, do the cut-off values should be adjusted in a similar way as the FDR_p-value or Bonferroni_p-value? Does this make sense?
1) The "n" vs. "n-1" distinction is not an issue. It's approximate either way. I used "n" in class because the formula is cleaner.
2) "We learned that the Chi-Squared gets larger as the sample size increases and this is not a desirable behavior of a fit index, right?"
Right.
"After reading the paper, I am not sure about that. In the article, it is mentioned several times that the RMSEA tends to over-reject with small sample sizes. So that means that it is sensible to the sample sizes, right?"
Well, yes, it is a statistic, and therefore random. Just like XBAR, it has higher variance with small n and smaller variance with large n (recall Var(XBAR) = sigma^2/n.) So, certainly sample size matters, but only in terms of variability of the statistic. Higher sample size means the statistic is more accurate; ie, closer to the true discrepancy measure.
" So I wonder if whenever I have small sample sizes should I focus more on the Chi-Square index, and with larger sample sizes should I focus more on the RMSEA index? And then, how large should n need to be in order to be considered large? "
I really dislike "sample size rules." They are usually silly. Like the old "n>30,use z" rule you used to see in textbooks, they are just silly.
The big picture is simply this: With smaller sample sizes, all estimates are less reliable. The entire analysis and conclusions are always more tenuous and uncertain when you have smaller sample sizes, rather than large. It's as simple as that. As far as fit statisitics go, you can use the confidence interval for RMSEA to express the uncertainty of model fit, for all sample sizes, but the interval is particulrly relevant for small sample sizes because it displays your uncertainty, due to small sample sizes, very clearly. In addition, the usaal t-stats and associated confidence intervals for path coefficients will accurately reflect you incertainty due to small smaple sizes. So, just use the confidence intervals, for any sample size. They will show you the range of uncertainty that is determined by your sample size.
And just to be absolutely clear: larger sample sizes are *always* better.
3) "Ho: S = S(q) may not be true if the manifest variables do not have a normal behavior and/or the estimated parameters are incorrect. Is this right? Then, it is not clear to me why the normality condition of the manifest variables is involved if the model being tested involves latent variables? Isn't the structure of S implied by the SEM model we are specifying? Should we check the normality assumption before any attempt of SEM analysis?"
Actually, the hypothesis Ho: S = S(q) itself has absolutely nothig to do with normality. All you need is the various conditions about terms bing uncorrelated. Recall that Cov(AX) = A*Sigma*A' is true, regardless of the normality of X. And the SEM form S(q) is strictly a consequence of Cov(AX) = A*Sigma*A' .
The issue they mention relates to the chi-square test - it is valid only under MVN. If data are not MVN, then the test is not valid. The degree of "non-validity" depends on the degree of non-normality. There are alternative tests that do not assume MVN, but the cure can be worse than the disease... In any event, one should think clearly about the source of non-MVN and act appropriately first. Eg, if the data are discrete, then simply use the polychoric corr. matrix. If there are gross outliers, might it make more sense to log-trnsform the data? The usual advice applies.
4) "My last question is concerned with the method of analysis. The authors used 84 experimental conditions and 500 replications for each condition. Then they did hypothesis testing. Well, we learned the problem implied with multiple hypothesis testing: the probability of committing at least one type I error increases. Thus, the more tests I perform the more probability of making a false conclusion. False rejections are easily observed on the graphs shown in the article. Shouldn't this issue be considered? In order to do that, do the cut-off values should be adjusted in a similar way as the FDR_p-value or Bonferroni_p-value? Does this make sense?"
Sure, it makes sense. But I thnk the authors are mode concerned with estimation than testing, even though the estimtes are themselves about tests. But I don't think it's an issue here. The concern would be that they conclude in some cases incorrectly, ie, maybe they conclude, incorrectly, that one method is better thn another in one of the 84 conditions. I am sure that such probably occured, after all, 500 is too small a sample size to estimate things precisely. But I don't think their main, overall conclusions are harmed by an occasional mistake of this type.
100 100 100 100