An abnormally high rate of cancer deaths among the wives and children of asbestos factory workers raises fears that household exposure to asbestos is far riskier than previously thought, an American Cancer Society researcher said Wednesday.

Dr. Lou Joubert, a staff scientist at the society, said she has studied 878 households since 1975 in which the head of the household worked in a New Jersey asbestos plant.''Among these wives and children, we have found that approximately 40 percent of the deaths to Jan. 1, 1990, were of cancer," Dr. Joubert said.

"We urge that studies of other groups of family contacts be undertaken so that additional data can be made available to confirm whether such limited asbestos contact is as dangerous as it now appears, and whether many more people may be at risk of asbestos-induced cancer than is now anticipated," Dr. Joubert said in a statement.

Her findings provide strong new evidence that people exposed to even very low levels of asbestos dust in the air may run a higher than normal risk of cancer.

In this case, family members apparently increased their risk simply by living in the same house with an asbestos worker and involuntarily breathing asbestos fibers brought home on the worker's clothing.

''Family members were only intermittently exposed, largely to fibers brought home on workers' clothing and thus serve as a paradigm for workers and building occupants who are now only intermittently exposed as asbestos- containing materials begin to disintegrate," said a statement by the Collegium Ramazzini, an international society of environmental and occupational health specialists.

The society is sponsoring a conference on asbestos disease in New York Thursday through Saturday, at which Dr. Joubert is scheduled to present her findings.

Dr. Joubert's study is certain to fuel an already-lively debate among asbestos experts over the extent to which asbestos used in schools, hospitals and other buildings must be cleaned up to avoid endangering occupants' health.

The debate is double-edged because the cleanup workers themselves increase their risk of disease by removing asbestos from areas where it might otherwise do no harm.

The asbestos was installed at a time when medical experts were unaware of the dangers of breathing the fibers.

School officials and building owners will spend billions of dollars in coming years to remove asbestos from ceilings and walls where it was used as insulation, a fire retardant or in other building materials.

Asbestos has also caused severe problems for some manufacturers, including Manville Corp., whose trust fund to pay for asbestos-related injuries is now the subject of a heated dispute.

Manville was driven into bankruptcy proceedings in 1982 because of asbestos-related claims. The Denver-based forest products and fiberglass company no longer makes the substance.

Some scientists have argued that the public is unduly panicked about asbestos fibers, maintaining that in many cases the fibers are best left in place or are of a type that pose little or no risk to humans.

In another study to be presented at the New York conference, Dr. L. Christine Oliver of the Harvard Medical School found that custodians in Boston public schools had abnormally high rates of scarring in the lining of the lung, a condition she attributed to job-related asbestos exposure.

In a telephone interview, Dr. Oliver said one-third of the 120 custodians she examined suffered from scarring of the lung linings. The longer they had worked as custodians, the morelikely it was that their lungs were scarred, she said.

Many of the custodians also suffered a reduction in the amount of air their lungs could take in, a condition she said may or may not be due to asbestos exposure.

Because the custodians were in some cases exposed to very high levels of asbestos, Dr. Oliver said she now wants to study teachers, whose exposures should be closer to those experienced by school children.

"What the findings in custodians say for students is hard to know at this point," Dr. Oliver said. "It's likely that teachers and students have lower exposures."