This shouldn’t be news at this point (JDK 1.5 was released 5 years ago), but in Java 5 Sun decided to deprecate large swaths of java.util.Date’s methods and constructors and introduce the Calendar class, which can handle locale-specific conversions and manipulations of the units of time we humans are used to seeing. This has created some confusion here and there about whether to represent datetimes as Dates or as Calendars, as it looks at a casual glance like Date has been more or less completely deprecated. It remains, however, as the preferred way of representing datetimes in Java. An explanation is after the jump.

Remember, it never says that Date itself is deprecated, only the methods of conversion are. Also, the javadoc for Calendar explicitly describes it as a way of converting between, not representing, different points in time relative to different calendar systems. The idiomatic usage pattern has a datetime being represented as a Date, but then converted to the locale-specific Calendar object to manipulate the date by saying “2 days from now” or “1 month ago”, which are locale-specific manipulations.

It seems like this is more complecated than it needs to be. Why not just keep datetimes as Calendars and be done with it? There’s a major problem with this approach, however. Since Calendar is abstract, one concrete calendar object may work very differently from another. For instance, say an entitiy (A) in Locale A computes and stores a Calendar object, and then another entity (B) in Locale B recieves the Calendar. If B attempts to say “one month from this point in time” using the Calendar from A, it will act according to A’s Locale, not B’s. Date does not have this problem, and is a representation of UTC. Datetimes are usually best represented as Date objects, and manipulated as Calendars, and then converted back to Dates for storage or communication between systems.

It would be nice in the future to be able to have convenience functions on Date to do Calendar-like things within whatever the current Locale is, but for now, we have to keep that distinction.

Go vote, if you haven’t already.

Get out the vote. Tell your friends, family, co-workers, fans, enemies, frenemies, random people on the street.

Find out where to vote: http://maps.google.com/vote

Today is the day to make history.

[Edited to add Obama's positions on NASA and Space Exploration]

Those of you who know me already know that I support Barack Obama for president. I’ve hesitated to post this here, due to the non-political nature of this blog. However, it is long past time for me to make the case for Obama and his policies on science and research. More after the cut on his key positions and why they are important.

Here are Obama’s key positions on science policy, with my commentary:

Restoring integrity to U.S. science policy to ensure that decisions that can be informed by science are made on the basis of the strongest possible evidence.

This is essential to any sane science policy. If we can learn more about an issue through evidence that has been scientifically validated, we will be much more effective in creating positive change. Science can inform policy in many ways, and this has not happened for 8 years, and wouldn’t happen under a McCain presidency.

Doubling over a 10 year period the federal investment in basic research by key science agencies, with a special emphasis on supporting young researchers at the beginning of their careers, and backing high-risk, high-return research.

High risk science has been at a standstill for decades as funding for science agencies has dried up. When funding is eliminated, funding agencies become more risk averse and focus funding on researchers who have already proved themselves. This often (but not always) leads to incremental improvements and a risk-averse environment for researchers who need to propose grants to fund new breakthroughs. Science is inherently risky, and no one knows what advances can be made. By increasing the ability of researchers to perform more risky, cutting edge research, we broaden our horizon of the possible with the dollars we spend.

Making a national commitment to science education and training by recruiting some of America’s best minds to teach K-12 math and science and by tripling the number of the National Science Foundation’s Graduate Research Fellowships.

As a computer scientist, I have no incentive to teach math or science to K-12 classrooms. I’m qualified to teach most of the math curriculum, any programming courses, or any basic science class. The people who do science are not teaching science to our children, which means that the quality of that education suffers, because teaching science in K-12 inevitably falls back on teaching “science facts”, rather than teaching students what falsifiability is, or how to set up an experiment with proper controls.

Also, the NSF’s Graduate Research Fellowship is a program to fund graduate students for Masters and Doctoral studies in the sciences. I know people who are on this fellowship, and they are doing amazing work. We need more of them.

Encouraging American innovation to flourish by making the R&D tax credit permanent, streamlining our patent system, eliminating the capital gains tax on start-ups and small businesses, and promoting the deployment of next-generation broadband networks.

I used to run a small technology startup. We need more R&D in this country, it and basic science are the engines that drive our economy. We have seen the amazing advances we can make when we encourage innovation, and we have seen the collapses that occur when we rest on our laurals and don’t drive innovation forward.

Addressing the “grand challenges” of the 21st century through accelerating the transition to a low-carbon, oil-free economy, enabling all Americans to live longer and healthier lives, and protecting our country from emerging threats to our national security.

Our carbon impact could easily be one of the worst things that have ever happened to the planet, causing mass extinctions and untold damage to the world’s ecosystem. But it can also be one of our greatest triumphs. Almost since life began, oxidation of carbon = energy. By using petroleum we simply accellerated the pace of that equation. By switching to technologies that can directly harness the power of the sun and earth, using wind, solar, hydro, geothermal, and renewable biofuels, we can “farm” energy rather than depending on “wild” depleted reserves. This can be done, but we need incentives to do this.

STRENGTHEN AMERICA’S LEADERSHIP IN SPACE SCIENCE

It is essential to maintain U.S. leadership in space.  President Bush set forth a bold mission for NASA, but failed to provide the leadership and funding to see it through.  As a result, in 2010, the United States will face a five-year gap in the ability to send astronauts to the International Space Station.  An Obama administration will restore America’s leadership in space science:

• Establish a robust program for human and robotic exploration that preserves our space workforce, engages international allies and draws on expertise in the private sector.
• Close the gap in NASA access to lower earth orbit and better utilize the International Space Station.
• Strengthen NASA’s missions in space science, weather, climate research, and aeronautical research.
• Develop a new generation of space vehicles to replace the Space Shuttle scheduled to retire in 2010.
• Improve NASA’s educational outreach and programs that promote spin-off consumer technologies.

There’s a lot more to Obama’s science plan. It is very comprehensive and evidence-driven. He will listen to scientists to find out what science can answer, and to find out what is needed to get those answers. I encourage you to read his plan, available here. Please remember to vote!

There are specific neural processes that take place when people meditate, which is demonstrated in the PLoS ONE paper “Thinking about Not-Thinking”: Neural Correlates of Conceptual Processing during Zen Meditation:

Recent neuroimaging studies have identified a set of brain regions that are metabolically active during wakeful rest and consistently deactivate in a variety the performance of demanding tasks. This “default network” has been functionally linked to the stream of thoughts occurring automatically in the absence of goal-directed activity and which constitutes an aspect of mental behavior specifically addressed by many meditative practices. Zen meditation, in particular, is traditionally associated with a mental state of full awareness but reduced conceptual content, to be attained via a disciplined regulation of attention and bodily posture. Using fMRI and a simplified meditative condition interspersed with a lexical decision task, we investigated the neural correlates of conceptual processing during meditation in regular Zen practitioners and matched control subjects. While behavioral performance did not differ between groups, Zen practitioners displayed a reduced duration of the neural response linked to conceptual processing in regions of the default network, suggesting that meditative training may foster the ability to control the automatic cascade of semantic associations triggered by a stimulus and, by extension, to voluntarily regulate the flow of spontaneous mentation.

Bible Cross-References: The bar graph that runs along the bottom represents all of the chapters in the Bible. Books alternate in color between white and light gray. The length of each bar denotes the number of verses in the chapter. Each of the 63,779 cross references found in the Bible is depicted by a single arc - the color corresponds to the distance between the two chapters, creating a rainbow-like effect.

This is a pretty cool visualization of bible cross-references that was developed by Chris Harrison to show bible cross references throughout the length of the bible. I can’t help but think of how interesting this could be for genome visualization: the cross references could be based on genes that are transcription factors for other genes. (activate or deactivate those genes). Chris has lots of other fascinating visualization projects to look at, and is very stimulating to browse it.

“no great genius was without a mixture of insanity”
- Aristotle

“They say madness runs in our family. Some even call me mad! And why? Because I dared to dream …of my own race of atomic monsters! Atomic supermen with octagonal-shaped bodies that suck blood out of…”
-Prof. Hubert Farnsworth

To be clear, by genius I mean the ability for humans to think at the level we do. And by madness, I mean madness. A recent study suggests that we have our big brains at a high price: schizophrenia. It comes down to the massive metabolism needs that our brains have. A team led by evolutionary biologist Philipp Khaitovich of the Shanghai Institutes for Biological Sciences and the Max Planck Institute for Evolutionary Anthropology created an experiment to see how much genes that are involved in schizophrenia have evolved since humans split from chimpanzees.

From “Madness: Price of a Big Brain?” (ScienceNOW):

First, the researchers looked at published databases of positively selected brain genes, which have been classified into 22 categories according to their function. They found that six of the categories included a high proportion of genes also implicated in schizophrenia; the genes in these six categories relate to energy metabolism.

So the team focused its search on energy pathways in the brain. Using a technique called nuclear magnetic resonance spectroscopy, the researchers measured the concentrations of 21 metabolites key to nerve function in the brains of 10 deceased schizophrenia patients and 12 normal human controls. Specifically, they examined an area of the prefrontal cortex implicated in social cognition. Nine of the metabolites, such as lactate, choline, and acetate, showed significantly different concentrations–some higher, some lower–in schizophrenics and normal humans. That finding, the authors say, confirms previous studies that brain metabolism is “substantially altered” in schizophrenia.

The researchers then looked at whether those nine metabolites might be important in human brain evolution. When they measured the concentrations in the same area in chimp brains, the team found that the differences between chimps and normal humans were much greater for those nine than for the 12 metabolites not implicated in schizophrenia, suggesting that energy pathways implicated in schizophrenia were also altered by human evolution, the team reports this week in Genome Biology. And 40 genes involved in these nine schizophrenia-related pathways also differed much more between chimps and humans than genes associated with the other 12.

We (my lab, http://krauthammerlab.med.yale.edu) recently published on our new image search engine for biomedical images, called YIF, or Yale Image Finder. From our blog post on our web site:

We have recently released a new biomedical image search engine we call YIF. You can access it at:

http://krauthammerlab.med.yale.edu/imagefinder

You can search the actual image content of over 34,000 Open Access articles from PubMed Central. We use OCR with different levels of image correction (article and corpus) for highly accurate image text extraction.

For more details about our algorithms, we have a paper in Bioinformatics titled “Yale Image Finder (YIF): a new search engine for retrieving biomedical images“.

15 hours and 40 doctors later, a farmer has two new arms. The surgery replaced both limbs of a farmer who lost his arms in an accident in 2002. He lost both arms below the shoulder. From the Guardian article:

Christoph Höhnke, a surgeon on the transplant team, said that the complicated procedure was completed without any unforeseen problems. It involved a team of 40 doctors, nurses and assistants working together, attaching one arm and then the other. “The whole thing went according to script,” he said.

With a surgery as complicated as this, though, many things can go wrong. The patient can reject the transplants, although close matching of blood and other immunological factors can minimize this, along with drugs that can dampen the immune system’s reaction to foreign bodies. The bones may not completely graft together. It’s like recovering from two severely broken arms on top of everything else.

Nerve regeneration has come a long way, however. The man who received a single arm transplant in 2006 was able to write with the transplanted hand within a year.

“nerve regeneration” at Yale Image Finder.

My friend Dan at Invisible Blocks came up with a great way to compute a long-running mean from the count and mean:

count += 1
mean += (x - mean) / count

I remembered that I had come up with a similar thing for standard deviation back when I was developing clustering algorithms that could use that value. It uses a power sum average, where you track the power sum as an average (divide the power sum by n) in a similar way.

n=0
mean=0
pwrSumAvg=0
stdDev=0

def update(x):
    n += 1
    mean += (x - mean) / n
    pwrSumAvg += ( x * x - pwrSumAvg) / n
    stdDev = sqrt( (pwrSumAvg * n - n * mean * mean) / (n - 1) )

We get this using the running sum formula for standard deviation and essentially simplifying away the sum by removing n:

In this formula, the core values for the standard deviation is the sum of the squares minus the square of the sum, which we can recreate from the mean and power sum average, respectively:

p: power sum average
m: mean
n: elements

Sorry for all the math, but I wanted to make sure I got it right, and that I can prove this later. I’ve already had to reconstruct it once, so I don’t want to do it again. Also, I just figured out how to embed equations in WordPress, which is pretty damn cool.

Another neat thing to do with this has to do with manipulating centroids of vectors. That is, dealing with lists of values that correspond to the same datum. For instance, a row of data can be considered an n-dimensional vector, where each value in the row is a value along that dimension of the vector. The traditional data mining definition of a centroid usually deals with only the mean, but I like to refer to centroid clouds when adding in standard deviations, because rather than a point in the n-dimensional space, we refer to a diffuse cloud of probability when adding in the standard deviation value.

To find a centroid of two value vectors, it is a simple matter to apply the formulas above to create the mean and standard deviations of the centroid by applying them to each value along the vector. Combining a centroid with a value works the same way, where the centroid is a vector of mean and power sum averages, and the count appies to the entire centroid.

A neat part comes in when you try to combine two centroids. The equations and code from before become  more symmetrical than when only adding in one value at a time. A given value, , can be generalized into a mean of and a count of . By doing this, we can change our code to be:

class value:
    _init_(self, x=0):
        self.n=1
        self.mean=x
        self.pwrSumAvg=x*x
        self.stdDev=0

    def merge(self, val):
        self.mean = (val.mean*val.n + self.mean*self.n)
                    / (self.n + val.n)
        pwrSumAvg = ( val.pwrSumAvg * val.n + self.pwrSumAvg * self.n)
                    / (self.n + val.n)
        self.n += val.n
        stdDev = sqrt( (pwrSumAvg * n - n * mean * mean) / (n - 1) )

We lose our nice calculation of the mean, but we gain the ability to merge these two centroids together. Note that the calculation for the standard deviation is the same as before, but we change the way the power sum average and the mean are calculated.

Why go through all this trouble? You can now use this for data clustering and also to potentially visualize the centroid clouds with their data scattered through them. In bioinformatics, for example, data is often summarized at many levels. Gene expression probes are grouped into probe sets, and probe sets are often clustered or combined into functional units, or gene sets. Preserving standard deviation at these levels allows for greater clarity into the data.

Data mining is, in the most general terms, an attempt to extract patterns and knowledge from data using various types of software and techniques. Data mining is used to learn and predict. This is applied to biology, neuroscience, fraud detection, national security, and even sports.

Some of these are more successful than others. For instance, text mining has been very successful at extracting proper nouns (names, places, etc.) from text, and what might be considered the biggest success of data mining comes from text mining: internet search engines. But at the same time, text mining has been less successful at automated text summarization.

Typically, data mining algorithms are broken into two groups: supervised and unsupervised. A supervised algorithm attempts to learn labels that have been applied to the data via some external means. Supervised algorithms can be used as either classifiers or predictors, based on whether the output of the algorithm (the labels) are either discrete (as in a classifier), or continuous (as in a predictor). An unsupervised algorithm tries to make some sense of the data without the use of labels. There are also other sorts of algorithms that don’t fit easily into these groups, such as optimizers. An optimizer attempts to optimize the parameters of an algorithm to maximize some sort of score in a multidimensional search space. Evolutionary Algorithms, such as genetic algorithms and genetic programming, along with Monte Carlo simulation and simulated annealing, are among the most common of optimization algorithms.

The inputs used in data mining are usually called features, and outputs are called labels, even if they are continuous values, and even if they aren’t being trained. An example is a row of data in the data set. Each example is a set of features and possibly labels. A model is the set of data that is the output of the analysis of the training data, and represents the algorithm’s “understanding” (as much as it can be called that) of the training data. A typical processing pipeline used in data mining will look like this:

• Feature Selection: Often done by the person running the analysis, it is critical to find features that have some chance of performing the task you set out. You can’t use a coin toss to predict the winner of a basketball game, no matter how many times you try. It can also be done at a later stage as an automated process called automated feature selection.
• Normalization: There are two kinds of normalization: corrective normalization, which is used to remove measurement artifacts (this is very common in bioinformatics), and algorithmic normalization, which is used to bring the values into a range that the data mining algorithm will understand. A common form of algorithmic normalization is to bring the mean of all values to 0 and the standard deviation to 1, or to bring the minimum to 0 and the maximum to 1. It really depends on the data and the algorithm. In fact, after feature selection, proper normalization is probably the most important factor to successful data mining.
• Training: Training is the creation of the data mining model for that data. In supervised cases, this is by comparing the training data with the associated labels, and attempting to predict those labels from the data. In unsupervised cases, a model is built up based on some sort of similarity of the cases.
• Testing: This usually only occurs in supervised learning. A set of examples are reserved for testing the accuracy of the data mining model. The classifier or predictor is run over the reserved examples, and the outputs are compared to the already-existing labels. The scoring can take a number of forms, which I hope to go over later. The most rigorous form of testing is called cross validation. It is done by randomly partitioning the data into N parts, training a model on N-1 parts, and then testing the model on the remaining part. This is repeated for all N parts, and the scores are combined across the data. The most extreme form of this is where N is the number of examples in the data set, and is called leave-one-out cross validation. Otherwise, it is called N-fold cross validation.

A common criticism of data mining has been that it is hypothesis fishing. That is, rather than observing the data, making a hypothesis, and testing the hypothesis, we let the computer examine all possible hypotheses and then pick the highest scoring one. I see data mining more as automated hypothesis generation. It is very difficult to get valid results from data mining without knowing the domain you are applying it to, and ultimately the model that gets created will need to be vetted against unlabeled data in some sort of prospective study. Once that happens, the resulting model can go from being a hypothesis to a theory.