Error in results[[elt]][[K_index, ll_index]] <- el(res, where = elt) : incompatible types (from NULL to list) in [[ assignment (aka NAs are not supported)

[Rahel14350]

Hi,
I have RRBS data from 44 Lung cancer samples (Ctrl and diseases). There is high heterogeneity beside the tumor cells in the samples, that is why after pre-processing with RnBeads, I submitted the Beta value matrix to MeDeCom. Now, I got this error, Can you please help me. Many thanks in advance,

>medecom.result<-runMeDeCom(persites_betaValue, Ks=3:10, lambdas=c(0,10^(-5:-1)), NINIT=10, NFOLDS=10, ITERMAX=300, NCORES=9)

[Main:] checking inputs
[Main:] preparing data
[Main:] preparing jobs
[Main:] 2768 factorization runs in total
[Main:] finished all jobs. Creating the object
Error in results[[elt]][[K_index, ll_index]] <- el(res, where = elt) :
incompatible types (from NULL to list) in [[ assignment
In addition: Warning messages:
1: In mclapply(rev(concurr_indices), function(index_group) { :
80 function calls resulted in an error
2: In mclapply(rev(concurr_indices), function(index_group) { :
8 function calls resulted in an error

[shmilyfhh]

I have the sample problem as well.

[lutsik]

This error message is unspecific, merely indicating that the worker threads have failed in a multicore setting. To get the actual exception, try rerunning with NCORES=1.

[nikitaved]

Could be a problem within a solver. Could you please share the dataset and/or your script?

[shmilyfhh]

Since we are working on large dataset, 40 cores took two days to get it done. I will try with one node, and see what is happening. By the way, does MeDeCom takes dataset with NAs? In my situation, it went wrong when my dataset has NAs. Thanks, huihui From: Pavlo Lutsik <notifications@github.com> Reply-To: lutsik/MeDeCom <reply@reply.github.com> Date: Monday, August 14, 2017 at 4:13 AM To: lutsik/MeDeCom <MeDeCom@noreply.github.com> Cc: "Fan, Huihui" <Huihui.Fan@vai.org>, Comment <comment@noreply.github.com> Subject: Re: [lutsik/MeDeCom] Error in results[[elt]][[K_index, ll_index]] <- el(res, where = elt) : incompatible types (from NULL to list) in [[ assignment (#6) This error message is unspecific, merely indicating that the worker threads have failed in a multicore setting. To get the actual exception, try rerunning with NCORES=1. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://secure-web.cisco.com/1u0hSYkn-mPPDGvUbwcu-CZhSOrg9r14avxPwmlLKuVURXapqIgHYPazm0uc7EQv_R24nClraEX5dRNhneJgiF5m0uWVpbemzsJWz6mKjn8IXaCR-lqXX28b6dmrH4zUnAKej4TQh8yrrk115quxQ4s9P2JuuYx2xt19Bt2XrK5QR02gf6T34AyHH119p6U5beKeeSPdzSfBs2U5AdR5RquYuQCmQVe_NXk4gsxzNZdPcQUu3CTfn-VaXiEH0y2vCJ7ojmDAMtGeB_cz7l9IoDw/https%3A%2F%2Fgithub.com%2Flutsik%2FMeDeCom%2Fissues%2F6#issuecomment-322126474>, or mute the thread<https://secure-web.cisco.com/1To9pSIMRqfIiZvOcha30KnWTZNYbTiCsjyS68stUGOaqAvX6jBvTmmnlNjHdEgOVlp4f8xwQn7cN_SfiDfgnUSoIRaDJr71JR9f8B_PyIROmRyyff9x_-nsDGL3eosNlqPZNtv9QUMJIpLZiwH4iCZxjbJSp_yy-go21oxb4ONfT3RNCt8kq_U50lpdX1eATNV55BZP4d2yFKqIbe4YtuXtLxEZv3ADnHn6wyGk8Pll7jhUL-DvLkqOz_sf3PNR6Q-45fjBT5W-KzyCfmB69bA/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQqqApFIhN98IQ8qUGqBUovQGtI7PVxAks5sYAFugaJpZM4OuwBw>.

[shmilyfhh]

What I did is just to use the example lines included in your manual. I run into errors with NAs included, but I don’t want to lose too much information, so I first filtered probes with NAs within more than half samples and then replace the remaining NAs with 0s. In order to reduce the computation time, I increase the core number to 40. Thanks, huihui From: nikitaved <notifications@github.com> Reply-To: lutsik/MeDeCom <reply@reply.github.com> Date: Monday, August 14, 2017 at 5:57 AM To: lutsik/MeDeCom <MeDeCom@noreply.github.com> Cc: "Fan, Huihui" <Huihui.Fan@vai.org>, Comment <comment@noreply.github.com> Subject: Re: [lutsik/MeDeCom] Error in results[[elt]][[K_index, ll_index]] <- el(res, where = elt) : incompatible types (from NULL to list) in [[ assignment (#6) Could be a problem within a solver. Could you please share the dataset and/or your script? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://secure-web.cisco.com/15GnpA4sNe7uWe_Ws0pyrPG3pNXym9-oKQnMauGJzCNztXk9_i12pAuSsxPvCAmke3MK4t8FgPf_yU2mFd3YNmAQllId6nKz-A5uocba71KWBVZXByYonqPAOUaKFHYgLP2xILKXPEcRWSoSM9BBDn26pCQQIEDRsWb3fK-7nUAuK0j13dlv3rSSCzEN0xajjAdsN4Ugz2ABPRqRkxPLks0S8Reg2TX0AZkwLd4noY7WiIOfCx_oRAfyyZ6s0xQFg4-SjGTWhiOgA8dytoxlcCA/https%3A%2F%2Fgithub.com%2Flutsik%2FMeDeCom%2Fissues%2F6#issuecomment-322149654>, or mute the thread<https://secure-web.cisco.com/1_fv_z0bIZKUjPlTApEoh5JHIJNExdciUpNgYW_44fPdHVw-ON0hdbaVU0wtBvm6ErOqyIKQhHcTWAOkT31WhrPo6MpG-dImuogxZiSyQBKNVP9C6Mq4LxLKmfUsT0AbVlM5jViIVSNDLwEvE5k_twPU_gu6RZTCX_clowTqxJ_JW-QIoo1g7QpqO3aKNXnkI3XXIC5yCE90Nj1N77mhkV1N-9jVN1VXiT1iNPGqPjdOfF67BYABvoXi58RskRITz3PVa2r3h5ejKP7cpYA7VRA/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQqqAsHaBgYOIgxDNCOGZBdqlD7Iq_Byks5sYBoXgaJpZM4OuwBw>.

[lutsik]

Regardless of decomposition, substituting NAs with zeroes is a bad idea anyway since 0 is one of the two extreme methylation values and will definitely lead to artefacts in your analysis.

I would suggest to reduce the data set to something minimal (several thousands of rows) and try running on one core to identify the source of your problem.

Did the test data set work fine for you?

[shmilyfhh]

Yes, the test data set works just fine both on one core and multiple cores. I agree with what you said, that was a bad decision. Sure, I could definitely try a subset of probes and go from there. Thanks, huihui From: Pavlo Lutsik <notifications@github.com> Reply-To: lutsik/MeDeCom <reply@reply.github.com> Date: Monday, August 14, 2017 at 9:52 AM To: lutsik/MeDeCom <MeDeCom@noreply.github.com> Cc: "Fan, Huihui" <Huihui.Fan@vai.org>, Comment <comment@noreply.github.com> Subject: Re: [lutsik/MeDeCom] Error in results[[elt]][[K_index, ll_index]] <- el(res, where = elt) : incompatible types (from NULL to list) in [[ assignment (#6) Regardless of decomposition, substituting NAs with zeroes is a bad idea anyway since 0 is one of the two extreme methylation values and will definitely lead to artefacts in your analysis. I would suggest to reduce the data set to something minimal (several thousands of rows) and try running on one core to identify the source of your problem. Did the test data set work fine for you? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://secure-web.cisco.com/19h4WRRmkCBrZCzlMRBG_OBXQ8djWsutFPlPF4Ixb-PRKoxnGlaRMEUmXrdD9yOuB3aMp0kS-MpugXQMnN7mIolJk321_QmL-rOZD8pSM3E4lIwIG6uW9AR0QMyqrnSr-n4Nr7BR1m4Sm4AhYJ1HN1Zm_yJlpEW7rvtiJ_flJTzf7OfvIQ85sxV-tyAD_MApC8ZcMHW82cPeOieZTv14eriOQlcBMrotkfTv70e7xFCiIvLskF5V_lEnJbBqdGkJnvE_xSZdGkRBWAJMRXig0GA/https%3A%2F%2Fgithub.com%2Flutsik%2FMeDeCom%2Fissues%2F6#issuecomment-322195808>, or mute the thread<https://secure-web.cisco.com/1GLpwAdWq7-3Nx4Xpa2dGEQxaebhl3v8WgaDSRRNL6ofMk0Itx2GxINbxQIwKQocqx9xZZHUGbDxRgYVH876t9GyKVjen7xbNv7ITJ0yzgnZk5bTIzp2nrejfkwGIFxhS_TS6N4NOlFTSOUNd40xFJSw_d2Y84N5jAYILc3xvsH2xnFlE2rW9-VKOLnLfb1-jcmwrVQenjzywvyZo2rcSi_t24T1rUmPOLA6pt_6LjjQsPau9lw0G2oTijn6x0Q_hziLGP2WGyQG3HQY8t1O0ew/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQqqAn_nU66HUYVrJlxk17ljpV2kdycIks5sYFErgaJpZM4OuwBw>.

[Rahel14350]

I also have problem with missing values and it is not even possible to get 1000 rows without NAs to run the test. Is there any solution for the NAs for example replacing them with 0.5? Or should I continue till the end of analysis with a small subset of my data (for example 500 rows)? ...
Many thanks in advance,

[shmilyfhh]

What I am doing with NAs is first I substitute them with 0s, and then I masked them as before with NAs after I got the results. Probes with 0.5ish methylation levels are considered as proof showing multiple cell compositions. I don’t think that is a good idea either. Not sure how Lutsik thinks though. Thanks, huihui From: RahelSh <notifications@github.com> Reply-To: lutsik/MeDeCom <reply@reply.github.com> Date: Monday, August 14, 2017 at 10:05 AM To: lutsik/MeDeCom <MeDeCom@noreply.github.com> Cc: "Fan, Huihui" <Huihui.Fan@vai.org>, Comment <comment@noreply.github.com> Subject: Re: [lutsik/MeDeCom] Error in results[[elt]][[K_index, ll_index]] <- el(res, where = elt) : incompatible types (from NULL to list) in [[ assignment (#6) I also have problem with missing values and it is not even possible to get 1000 rows without NAs to run the test. Is there any solution for the NAs for example replacing them with 0.5? Or should I continue till the end of analysis with a small subset of my data (for example 500 rows)? ... Many thanks in advance, — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://secure-web.cisco.com/1tcS8eQ8clzA9sfP-oHYsgMmVvRvArmx5PWpniMJTQSMSCrM5EjElz9FN67JkB5fddBK3Dx_xns9Mi5HBKGDWwnkBgrQDKgHD7Q2i7fWJacCpwaP1yx0JQYqebO_Cx7aK24lcGgR_j5WeHS4dTtRp9wxkpqUlRsc6s4EZ8Zc9YVRH4aWo7TEj8i1YhCfG-abEL1mfOoeMSemVM-gRa1gqI_wuk6rUNgTdmy2MFn12De9FZQQ07xBMRwrTd2n45KsBOQ4hen2KVL67Dhx8yoQZeQ/https%3A%2F%2Fgithub.com%2Flutsik%2FMeDeCom%2Fissues%2F6#issuecomment-322199265>, or mute the thread<https://secure-web.cisco.com/1A51D76miuIU--90raa1fcW9SHKZloNKqCQA1VwEXHD_7pk2SYExPXSMG_KtC7DYqAqnfGx6QJzzCAKww9nJEwdHWL29C4Goq9cLrsWgbMwtf1lZ5z_R0wjZeVOv4G-ihjecRa7h9nHOudUn6dVPa7GVohtyGd5RibJxmHEsExa_cVB3TXDJAACrRTcOGcMdxtK6LnvTYYl3nMRzbCAoUyjMniq_xLyRM0HixeAp94nBGfarhg8COBO1e2XBWJpx6kcTLeEJsy_Lb832u0rCujA/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQqqAjGE-ltdBB_85vo-BuczgmcsWw1wks5sYFQ6gaJpZM4OuwBw>.

[nikitaved]

The solver does not support NAs, that is for sure.

[Rahel14350]

For me, also when I replace the NAs value with some number, everything is working on MeDeCom, but I know that this is wrong. At the end there is no solution for NAs? If I continue with only a subset of my methylation values then is the analysis and output correct?

[nikitaved]

No, NAs are not supported as of now. Shall we just ignore their contribution in the loss function? We will check how people solve the task of missing values, and, if it is not that much of a work, we will add this feature once we have a bit more time... But I certainly agree it is worth doing.

[lutsik]

As a temporary solution, I would recommend looking into the imputation methods in case removing all NAs seems like an overkill.

In fact I started implementing something along these lines a while ago (see argument na.values and its usage in onerun.alternate, which is factorisations.R:445 and below). The idea was to disregard NAs when calculating the loss, and exclude NA-containing rows for the T and A updates. Not sure this was the right way, though.

[nikitaved]

In fact I started implementing something along these lines a while ago (see argument na.values and its usage in onerun.alternate, which is factorisations.R:445 and below). The idea was to disregard NAs when calculating the loss, and exclude NA-containing rows for the T and A updates. Not sure this was the right way, though.
I think it would not be that much different from simply removing NAs from a dataset (maybe slower convergence but smaller matrices)... I think we could develop an algorithm to solve an adopted version of the matrix completion problem (0-1 constraints) and then apply MeDeCom...