A tree-based method for the rapid screening of chemical fingerprints
© Kristensen et al. 2010
Received: 28 July 2009
Accepted: 4 January 2010
Published: 4 January 2010
The fingerprint of a molecule is a bitstring based on its structure, constructed such that structurally similar molecules will have similar fingerprints. Molecular fingerprints can be used in an initial phase of drug development for identifying novel drug candidates by screening large databases for molecules with fingerprints similar to a query fingerprint.
In this paper, we present a method which efficiently finds all fingerprints in a database with Tanimoto coefficient to the query fingerprint above a user defined threshold. The method is based on two novel data structures for rapid screening of large databases: the kD grid and the Multibit tree. The kD grid is based on splitting the fingerprints into k shorter bitstrings and utilising these to compute bounds on the similarity of the complete bitstrings. The Multibit tree uses hierarchical clustering and similarity within each cluster to compute similar bounds. We have implemented our method and tested it on a large real-world data set. Our experiments show that our method yields approximately a three-fold speed-up over previous methods.
Using the novel kD grid and Multibit tree significantly reduce the time needed for searching databases of fingerprints. This will allow researchers to (1) perform more searches than previously possible and (2) to easily search large databases.
When developing novel drugs, researchers are faced with the task of selecting a subset of all commercially available molecules for further experiments. There are more than 8 million such molecules available , and it is not feasible to perform computationally expensive calculations on each one. Therefore, the need arises for fast screening methods for identifying the molecules that are most likely to have an effect on a given disease. It is often the case that a molecule with some effect is already known, e.g. from an already existing drug. An obvious initial screening method presents itself, namely to identify the molecules which are similar to this known molecule. To implement this screening method one must decide on a representation of the molecules and a similarity measure between representations of molecules. Several representations and similarity measures have been proposed [2–4]. We focus on molecular fingerprints. A fingerprint for a given molecule is a bitstring of size N which summarises structural information about the molecule . Fingerprints should be constructed such that if two fingerprints are very similar, so are the molecules which they represent. There are several ways of measuring the similarity between fingerprints . We focus on the Tanimoto coefficient, which is a normalised measure of how many bits two fingerprints share. It is 1.0 when the fingerprints are the same, and strictly smaller than 1.0 when they are not. Molecular fingerprints in combination with the Tanimoto coefficient have been used successfully in previous studies .
We focus on the screening problem of finding all fingerprints in a database with Tanimoto coefficient to a query fingerprint above a given threshold, e.g. 0.9. Previous attempts have been made to improve the query time. One approach is to reduce the number of fingerprints in the database for which the Tanimoto coefficient to the query fingerprint has to be computed explicitly. This includes storing the fingerprints in the database in a vector of bins , or in a trie like structure , such that searching certain bins, or parts of the trie, can be avoided based on an upper-bound on the Tanimoto coefficient between the query fingerprint and all fingerprints in individual bins or subtries. Another approach is to store an XOR summary, i.e. a shorter bitstring, of each fingerprint in the database, and use these as rough upper bounds on the maximal Tanimoto coefficients achievable, before calculating the exact coefficients .
In this paper, we present an efficient method for the screening problem, which is based on an extension of an upper bound given in  and two novel tree based data structures for storing and retrieving fingerprints. To further reduce the query time we also utilise the XOR summary strategy . We have implemented our method and tested it on a realistic data set. Our experiments clearly demonstrate that it is superior to previous strategies, as it yields a three-fold speed-up over the previous best method.
2.1 kD grid
This bound can be used to speed up the search, by storing the database of fingerprints in N + 1 buckets such that bitstring B is stored in the |B|th bucket. When searching for bitstrings similar to a query bitstring A it is sufficient to examine the buckets where S max ≥ S min.
where X·Y is the concatenation of bitstrings X and Y .
where is a bound on the number of 1-bits in the logical and in the first part of the bitstrings. is a bound for the logical or in the first part of the bitstrings.
Similarly, is a bound on the last part.
2.2 Singlebit tree
The Singlebit tree is a binary tree which stores the fingerprints of a single bucket from a kD grid. At each node in the tree a position in the bitstring is chosen. All fingerprints with a zero at that position are stored in the left subtree while all those with a one are stored in the right subtree. This division is continued recursively until all the fingerprints in a given node are the same. When searching for a query bitstring A in the tree it now becomes possible, by comparing A to the path from the root of the tree to a given node, to compute an upper bound on S T (A, B) for every fingerprint B in the subtree of that given node. Given two bitstring A and B let M ij be the number of positions where A has an i and B has a j. There are four possible combinations of i and j, namely M 00, M 01, M 10 and M 11.
The path from the root of a tree to a node defines lower limits m ij on M ij for every fingerprint in the subtree of that node. Let u ij denote the unknown difference between M ij and m ij , that is u ij = M ij - m ij .
Remember that is known when processing a given bucket.
The Singlebit tree can also be used to store all the fingerprints in the database without a kD grid. In this case, however, |B| is no longer available and thus the bound cannot be used. A less tight bound can be formulated, but experiments, not included in this paper, indicate that this is a poor strategy.
2.3 Multibit tree
The experiments in Sec. 3 unfortunately show that using the kD grid combined with Singlebit trees decreases performance compared to using the kD grid and simple lists. The fingerprints used in our experiments have a length of 1024 bits. In our experiments no Singlebit tree was observed to contain more the 40,000 fingerprints. This implies that the expected height of the Singlebit trees is no more than 15 (as we aim for balanced trees cf. above). Consequently, the algorithm will only obtain information about 15 out of 1024 bits before reaching the fingerprints. A strategy for obtaining more information is to store a list of bit positions, along with an annotation of whether each bit is zero or one, in each node. The bits in this list are called the match-bits.
The Multibit tree is an extension of the Singlebit tree, where we no longer demand that all children of a given node are split according to the value of a single bit. In fact we only demand that the data is arranged in some binary tree. The match-bits of a given node are computed as all bits that are not a match-bit in any ancestor and for which all fingerprints in the leaves of the node have the same value. Note that a node could easily have no match-bits. When searching through the Multibit tree, the query bitstring A is compared to the match-bits of each visited node and m 00, m 01, m 10 and m 11 are updated accordingly. is computed the same way as and only branches for which ≥ S min are visited.
We have implemented the kD grid and the Single- and Multibit tree in Java. The implementation along with all test data is available at
Using these implementations, we have constructed several search methods corresponding to the different combinations of the data structures. We have examined the kD grid for k = 1, 2, 3 and 4, where the fingerprints in the buckets are stored in a simple list, a Singlebit tree or a Multibit tree. For purposes of comparison, we have implemented a linear search strategy, that simply examines all fingerprints in the database. We have also implemented the strategy of "pruning using the bit-bound approach first, followed by pruning using the difference of the number of 1-bits in the XOR-compressed vectors, followed by pruning using the XOR approach" from . This strategy will hereafter simply be known as Baldi. A trick of comparing the XOR-folded bitstrings  immediately before computing the true Tanimoto coefficient, is used in all our strategies to improve performance. The length of the XOR summary is set to 128, as suggested in . An experiment, not included in this paper, confirmed that this is indeed the optimal size of the XOR fingerprint. We have chosen to reimplement related methods in order to make an unbiased comparision of the running times independent of programming language differences.
The experiments were performed on an Intel Core 2 Duo running at 2.5 GHz and with 2 GB of RAM. Fingerprints were generated using the CDK fingerprint generator  which has a standard fingerprint size N of 1024. One molecule timed out and did not generate a fingerprint. We have performed our tests on different sizes of the data set, from 100,000 to 2,000,000 fingerprints in 100,000 increments. For each data set size, the entire data structure is created. Next, the first 100 fingerprints in the database are used for queries. We measure the query time and the space consumption.
Figure 7 shows the average query time for the different strategies and different values of k plotted against the database size. We note that the Multibit tree in a 1D grid is best for all sizes. Surprisingly, the simple list, for an appropriately high value of k, is faster than the Singlebit tree, yet slower than the Multibit tree. This is probably due to the fact that the Singlebit trees are too small to contain sufficient information for an efficient pruning: the entire tree is traversed, which is slower than traversing the corresponding list implementation. All three approaches (List, Singlebit- and Multibit trees) are clearly superior to the Baldi approach, which in turn is better than a simple linear search (with the XOR folding trick).
The reason why linear search is not constant time for a constant data set is that, while it will always visit all fingerprints, the time for visiting a given fingerprint is not constant due to the XOR folding trick.
The results seems to be consistent with the average query time presented in Fig. 10.
In this paper we have presented a method for finding all fingerprints in a database with Tanimoto coefficient to a query fingerprint above a user defined threshold. Our method is based on a generalisation of the bounds developed in  to multiple dimensions. Our generalisation results in a tighter bound, and experiments indicate that this results in a performance increase. Furthermore, we have examined the possibility of utilising trees as secondary data structures in the buckets. Again, our experiments clearly demonstrate that this leads to a significant performance increase.
Our methods allow researchers to search larger databases faster than previously possible. The use of larger databases should increase the likelihood of finding relevant matches. The faster query times decreases the effort and time needed to do a search. This allow more searches to be done, either for more molecules or with different thresholds S min on the Tanimoto coefficient. Both of these features increase the usefulness of fingerprint based searches for the researcher in the laboratory.
Our method is currently limited by the rather larger memory consumption of the Multibit tree. Another implementation might remedy this situation somewhat. Otherwise we suggest an I/O efficient implementation where the tree is kept on disk.
To increase the speed of our method further we are aware of two approaches. Firstly, the best way to construct the Multibit trees remain uninvestigated. Secondly, a tighter coupling between the Multibit tree and the kD grid would allow us to use grid information in the Multibit tree: in the kD grid we have information about each fragment of the fingerprints which is not used in the current tree bounds.
- Irwin JJ, Shoichet BK: ZINC: A Free Database of Commercially Available Compounds for Virtual Screening. Journal of Chemical Information and Modeling 2005, 45:177–182.PubMedView Article
- Gillet VJ, Willett P, Bradshaw J: Similarity Searching Using Reduced Graphs. Journal of Chemical Information and Computer Sciences 2003,43(2):338–345.PubMed
- Leach AR, Gillet VJ: An Introduction to Chemoinformatics. Kluwer Academic Publishers, Dordrecht, The Netherlands, rev. ed; 2007.View Article
- Willett P: Similarity-based approaches to virtual screening. Biochem Soc Trans 2003,31(Pt 3):603–606.PubMed
- Willett P, Barnard JM, Downs GM: Chemical Similarity Searching. Journal of Chemical Information and Computer Sciences 1998,38(6):983–996.
- Swamidass SJ, Baldi P: Bounds and Algorithms for Fast Exact Searches of Chemical Fingerprints in Linear and Sublinear Time. Journal of Chemical Information and Modeling 2007,47(2):302–317.PubMedView Article
- Smellie A: Compressed Binary Bit Trees: A New Data Structure For Accelerating Database Searching. Journal of Chemical Information and Modeling 2009,49(2):257–262.PubMedView Article
- Baldi P, Hirschberg DS, Nasr RJ: Speeding Up Chemical Database Searches Using a Proximity Filter Based on the Logical Exclusive OR. Journal of Chemical Information and Modeling 2008,48(7):1367–1378.PubMedView Article
- Saitou N, Nei M: The neighbor-joining method: a new method for reconstructing phylogenetic trees. Mol Biol Evol 1987,4(4):406–425.PubMed
- Steinbeck C, Han Y, Kuhn S, Horlacher O, Luttmann E, Willighagen E: The Chemistry Development Kit (CDK): An Open-Source Java Library for Chemo- and Bioinformatics. Journal of Chemical Information and Computer Sciences 2003,43(2):493–500.PubMed
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.