Vector field #33022

mayya-sharipova · 2018-08-21T12:37:14Z

Introduce dense and sparse vector fields that can be used for documents scoring.
This PR only introducing indexing of these fields.
Later PR will add script functions that can calculate a distance between
a given query vector and the indexed document vector.

1. Dense vector

PUT dindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_vector": {
          "type": "dense_vector"
        },
        "my_text" : {
          "type" : "keyword"
        }
      }
    }
  }
}

PUT dinex/_doc/1
{
  "my_text" : "text1",
  "my_vector" : [ 0.5, 10, 6 ]
}

2. Sparse vector

PUT sindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_vector": {
          "type": "sparse_vector"
        },
        "my_text" : {
          "type" : "keyword"
        }
      }
    }
  }
}

PUT sindex/_doc/1
{
  "my_text" : "text1",
  "my_vector" : {"1": 0.5, "99": -0.5,  "5": 1}
}

3 Implementation details

3.1 Dense Vector

BinaryDocValuesField
byte array ->
- array of integers (encoded array of float values)

3.2 Sparse Vector

BinaryDocValuesField
byte array ->
- array of integers (encoded array of float values)
- array of two bytes (encoded array of integer dimensions)

Relates to #31615

1. Dense vector PUT dindex { "mappings": { "_doc": { "properties": { "my_vector": { "type": "dense_vector" }, "my_text" : { "type" : "keyword" } } } } } PUT dinex/_doc/1 { "my_text" : "text1", "my_vector" : [ 0.5, 10, 6 ] } PUT dindex/_doc/2 { "my_text" : "text2", "my_vector" : [ 0.5, 10, 10] } GET dindex/_search { "query" : { "vector" : { "field" : "my_vector", "query_vector": [ 0.5, 10, 10] } } } Result: .... "hits": [ { "_index": "dindex", "_type": "_doc", "_id": "2", "_score": 1.0000001, "_source": { "my_text": "text1", "my_vector": [ 0.5, 10, 10 ] } }, { "_index": "dindex", "_type": "_doc", "_id": "1", "_score": 0.97016037, "_source": { "my_text": "text1", "my_vector": [ 0.5, 10, 6 ] } } ] 2. Sparse vector PUT sindex { "mappings": { "_doc": { "properties": { "my_vector": { "type": "sparse_vector" }, "my_text" : { "type" : "keyword" } } } } } PUT sindex/_doc/1 { "my_text" : "text1", "my_vector" : {"1": 0.5, "99": -0.5, "5": 1} } PUT sindex/_doc/2 { "my_text" : "text2", "my_vector" : {"103": 0.5, "4": -0.5, "5": 1} } GET sindex/_search { "query" : { "vector" : { "field" : "my_vector", "query_vector": {"99": -0.5, "1": 0.5, "5": 1} } } } Result: "hits": [ { "_index": "sindex", "_type": "_doc", "_id": "1", "_score": 0.99999994, "_source": { "my_text": "text1", "my_vector": { "1": 0.5, "99": -0.5, "5": 1 } } }, { "_index": "sindex", "_type": "_doc", "_id": "2", "_score": 0.6666666, "_source": { "my_text": "text2", "my_vector": { "103": 0.5, "4": -0.5, "5": 1 } } } ] Search with filter: GET sindex/_search { "query": { "bool": { "must" : { "match": { "my_text": "text2" } }, "should" : { "vector" : { "field" : "my_vector", "query_vector": {"99": -0.5, "1": 0.5, "5": 1} } } } } } Result: "hits": [ { "_index": "sindex", "_type": "_doc", "_id": "2", "_score": 0.6931472, "_source": { "my_text": "text2", "my_vector": { "103": 0.5, "4": -0.5, "5": 1 } } } ] 3. Implementation details 3.1 Dense Vector - BinaryDocValuesField - byte array -> - integer (number of dimensions) - array of integers (encoded array of float values) 3.2 Sparse Vector - BinaryDocValuesField - byte array -> - integer (number of dimenstions) - array of integers (encoded array of float values) - array of integers (array of integer dimensions) Relates to elastic#31615

mayya-sharipova · 2018-08-21T12:39:58Z

@jpountz I would appreciate your feedback on the API and implementation details.
This is WIP, it doesn't contain tests and documentation, the goal is to get feedback.

elasticmachine · 2018-08-21T12:40:21Z

Pinging @elastic/es-search-aggs

jpountz · 2018-08-23T08:13:16Z

The field looks ok to me. The thing that I'm unsure about is exposing this vector query: this query doesn't care much about matching, only about scoring. If you want to find the top documents sorted by cosine similarity across the whole index, Elasticsearch will run a linear scan. This is why I initially suggested exposing it as a rescorer instead. If the concern is that rescorers can't run on all hits, maybe there are other options we could think about like having some companion utility of function_score (or its replacement(s)) which would allow to easily fold cosine similarity into the score?

Remove vector query Add documentation Add tests

mayya-sharipova · 2018-11-12T17:37:59Z

@jpountz Hi Adrien, I tried to address your feedback with new commits. I have removed vector_query, and this PR is only concerned with introducing two new field types: dense_vector and sparse_vector. Later I plan to introduce functions to script score query that will do distance calculations based on these fields.

Can you please continue to review when you have available time.

jpountz

Thanks @mayya-sharipova. I did a first round and left some questions.

jpountz · 2018-11-20T20:43:07Z

docs/reference/mapping/types/dense-vector.asciidoc

+Number of dimensions in a vector can be arbitrary and can be
+different across documents.
+
+These vectors can be used for document scoring.


should we say _re_scoring?

I intended to use this field in scoring through script score functions. Do you think it qualifies for "scoring", or is still "re-scoring"?

OK let's keep "scoring".

ok let's keep scoring

jpountz · 2018-11-20T20:44:28Z

docs/reference/mapping/types/dense-vector.asciidoc

+=== Dense vector datatype
+
+A `dense_vector` field indexes dense vectors of float values.
+Number of dimensions in a vector can be arbitrary and can be


Let's put an upper limit on the number of dimensions and document it?

Do we aim to support multi-valued fields?

jpountz · 2018-11-20T20:44:52Z

docs/reference/mapping/types/dense-vector.asciidoc

+[[dense-vector]]
+=== Dense vector datatype
+
+A `dense_vector` field indexes dense vectors of float values.


"indexes" might be a bit misleading since they are not searchable?

jpountz · 2018-11-20T20:55:38Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+
+        @Override
+        public DocValueFormat docValueFormat(String format, DateTimeZone timeZone) {
+            return DocValueFormat.BINARY;


let's build a custom format that returns a float[] and test docvalue_fields with such fields? Or alternatively raise an error for now saying this is not supported. I don't like using the BINARY format because this will return a base64 string of the internal representation of this vector. And that would be a breaking change if we ever decided to actually support docvalue_fields on such fields.

jpountz · 2018-11-20T21:08:25Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+                if (buf.length < (offset + INT_BYTES)) {
+                    buf = ArrayUtil.grow(buf, (offset + INT_BYTES));
+                }
+                NumericUtils.intToSortableBytes(Float.floatToIntBits(value), buf, offset);


do we need sortability or are we using this existing method for convenience?

jpountz · 2018-11-20T21:12:10Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+                throw new IllegalArgumentException("[dense_vector] field takes an array of floats, but got unexpected token " + token);
+            }
+        }
+        NumericUtils.intToSortableBytes(dim, buf, 0); //recording number of dimensions at the beginning


do we need this? we could figure out the number of dimensions by dividing the array length by 4?

jpountz · 2018-11-20T21:14:10Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+    @Override
+    public void parse(ParseContext context) throws IOException {
+        if (context.externalValueSet()) {
+            throw new IllegalArgumentException("[sparse_vector] field can't be used in multi-fields");


jpountz · 2018-11-20T21:16:39Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+                throw new IllegalArgumentException("[sparse_vector] field takes an object that maps a dimension number to a float, " +
+                    "but got unexpected token [" + token + "]");
+            }
+        }


this performs in O(n^2) it seems. Should we collect values in the order in which they occur in the array and sort in the end?

However I agree that we should store dimensions in order

jpountz · 2018-11-20T21:18:48Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+        }
+        NumericUtils.intToSortableBytes(dim, buf, 0); //recording number of dimensions at the beginning
+        BinaryDocValuesField field = new BinaryDocValuesField(fieldType().name(), new BytesRef(buf, 0, offset));
+        context.doc().addWithKey(fieldType().name(), field);


It seems like this shouldn't support multi-valued fields. You could do that by failing if context.doc().getByKey is not null.

jpountz · 2018-11-20T21:21:56Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+                } catch (NumberFormatException e) {
+                    throw new IllegalArgumentException("[sparse_vector]'s dimensions should be integers represented as strings, but got ["
+                        + context.parser().currentName() + "]");
+                }


should we fail negative dimensions?

Also I'm wondering whether users would need more than eg. 65k dimensions. If we decide that not then we could encode the dimension on two bytes only.

mayya-sharipova · 2018-11-29T19:48:11Z

@jpountz Adrien, thanks for your feedback. I have tried to address all your comments, can you please continue to review when you have available time? Thank you

jtibshirani

I was just catching up on this (very exciting) work and had a couple suggestions.

jtibshirani · 2018-11-30T02:02:58Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+                        "[dense_vector] field has exceeded the maximum allowed number of dimensions of :[" + MAX_DIMS_COUNT + "]");
+                }
+            } else {
+                throw new IllegalArgumentException("[dense_vector] field takes an array of floats, but got unexpected token " + token);


Small comment: using XContentParserUtils#ensureExpectedToken could be nice here and in the check below, as it will include the location where parsing failed in the error message.

jtibshirani · 2018-11-30T02:12:39Z

.../mapper-extras/src/test/java/org/elasticsearch/index/mapper/DenseVectorFieldMapperTests.java

+
+        // assert that after decoding the indexed value is equal to expected
+        BytesRef vectorBR = ((BinaryDocValuesField) fields[0]).binaryValue();
+        float[] decodedValues = DenseVectorFieldMapper.decodeVector(vectorBR);


I wonder if it'd make sense to have dedicated unit tests for vector encoding + decoding, as the logic is fairly subtle? With a unit test (as opposed to an integration test like this one), you could test many cases quickly, and perhaps even use randomized inputs.

jtibshirani · 2018-11-30T02:14:00Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+
+    // insert dim into the sorted array of dims
+    // insert value in the array of values in the same position as the corresponding dim
+    public static void sortedInsertDimValue(float[] values, int[] dims, float value, int dim, final int dimIndex) {


Small comment: it could be nice to pull these methods into a dedicated class like SparseVectorEncoder to separate concerns a bit and allow for easy unit testing.

jpountz

I left some more comments @mayya-sharipova.

jpountz · 2018-11-30T15:44:39Z

docs/reference/mapping/types/dense-vector.asciidoc

+A `dense_vector` field stores dense vectors of float values.
+The maximum number of dimensions that can be in a vector should
+not exceed 500. The number of dimensions can be
+different across documents.


Let's mention it only supports single-valued fields? Ie. you can't do

{ "my_vector": [ [1, 2], [3, 4] ] }

or even

{ "my_object": [ { "my_vector": [1,2] }, { "my_vector": [3,4] } ] }

jpountz · 2018-11-30T16:35:06Z

docs/reference/mapping/types/dense-vector.asciidoc

+
+Internally, each document's dense vector will be encoded as a binary
+doc value. Its size in bytes is equal to
+`INT_BYTES * NUMBER_OF_DIMENSIONS`,


let's directly say 4 * NUMBER_OF_DIMENSIONS?

jpountz · 2018-11-30T16:40:07Z

docs/reference/mapping/types/dense-vector.asciidoc

+Number of dimensions in a vector can be arbitrary and can be
+different across documents.
+
+These vectors can be used for document scoring.


ok let's keep scoring

jpountz · 2018-11-30T16:41:17Z

docs/reference/mapping/types/sparse-vector.asciidoc

+
+Internally, each sparse document vector will be encoded as a binary
+doc value with the size in bytes equal to
+`(INT_BYTES * 2) * NUMBER_OF_DIMENSIONS`,


let's say directly 6 * NUMBER_OF_DIMENSIONS?

jpountz · 2018-11-30T16:49:11Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+                    dim = Integer.parseInt(context.parser().currentName());
+                    if (dim < 0 || dim > MAX_DIMS_NUMBER) {
+                        throw new IllegalArgumentException("[sparse_vector]'s dimension number must be " +
+                            "a non-negative integer value not exceeding [" + MAX_DIMS_NUMBER + "]");


let's provide dim in the message?

jpountz · 2018-11-30T16:49:36Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+                    }
+                } catch (NumberFormatException e) {
+                    throw new IllegalArgumentException("[sparse_vector]'s dimensions should be integers represented as strings, but got ["
+                        + context.parser().currentName() + "]");


can you add e as a cause of this exception?

jpountz · 2018-11-30T16:53:11Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+    private static BytesRef encodeSparseVector(float[] values, int[] dims, int dimCount) {
+        // 1. Sort dimensions in the ascending order and sort values in the same order as their corresponding dimensions
+        // as we expect that data should be already provided as sorted by dim,
+        // we expect just a single pass in this bubble sort algorithm


this exceptation is maybe too strong given that hashs are not supposed to be ordered in JSON. FYI Lucene has utility classes to sort parallel arrays, see eg. InPlaceMergeSorter, which also runs in linear time if the input is already sorted.

jpountz · 2018-11-30T16:57:49Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+
+    public static final String CONTENT_TYPE = "sparse_vector";
+    public static int MAX_DIMS_COUNT = 500; //maximum allowed number of dimensions
+    public static int MAX_DIMS_NUMBER = 65000; //maximum allowed dimension's number


let's make it 65536? (1<<16)

jpountz · 2018-11-30T17:00:41Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/SparseVectorFieldMapper.java

+            buf[offset] = (byte) (dims[dim] >>  8);
+            buf[offset+1] = (byte) dims[dim];
+            offset += 2;
+        }


not a big deal but it feels a bit weird to me to encode values before dimensions, it's a bit like encoding a value before its key?

mayya-sharipova · 2018-11-30T20:37:00Z

@jpountz @jtibshirani Thanks Adrien and Julie for the feedback. I have addressed it in the last commit. Can you please continue to review when you have available time?

jpountz

Thanks @mayya-sharipova, I left some comments mostly around adding the field name to error messages, but it looks good to me overall.

jpountz · 2018-12-06T16:06:44Z

modules/mapper-extras/build.gradle

@@ -20,4 +20,4 @@
 esplugin {
    description 'Adds advanced field mappers'
    classname 'org.elasticsearch.index.mapper.MapperExtrasPlugin'
-}
+}


let's undo changes to this file?

jpountz · 2018-12-06T16:07:38Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+/**
+ * A {@link FieldMapper} for indexing a dense vector of floats.
+ */
+


remove empty line?

jpountz · 2018-12-06T16:09:02Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+    private DenseVectorFieldMapper(String simpleName, MappedFieldType fieldType, MappedFieldType defaultFieldType,
+                                   Settings indexSettings, MultiFields multiFields, CopyTo copyTo) {
+        super(simpleName, fieldType, defaultFieldType, indexSettings, multiFields, copyTo);
+        assert fieldType.indexOptions().compareTo(IndexOptions.DOCS_AND_FREQS) <= 0;


hmm we could actually assert that fieldType.indexOptions() == IndexOptions.NONE since this field is not indexed?

jpountz · 2018-12-06T16:26:15Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+
+        @Override
+        public DocValueFormat docValueFormat(String format, DateTimeZone timeZone) {
+            throw new UnsupportedOperationException("[dense_vector] field doesn't support doc values");


A better message might be to say that it doesn't support docvalue_fields and aggregations (which are the two features that use this API. Let's also include the field name in the message?

jpountz · 2018-12-06T16:30:33Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+
+    public static final String CONTENT_TYPE = "dense_vector";
+    public static short MAX_DIMS_COUNT = 500; //maximum allowed number of dimensions
+    private static final byte INT_BYTES = 4;


let's remove this constant and use Integer.BYTES instead?

Thanks for the comment, Adrien! I was thinking since we have already precoded 4 and 2 bytes calculations as below, wouldn't it be safer to keep these constants as 4 and 2.

buf[offset] = (byte) (intValue >> 24); buf[offset+1] = (byte) (intValue >> 16);

But if it is fine, I can do as you suggested.

jpountz · 2018-12-06T16:43:07Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/VectorEncoderDecoder.java

+import org.apache.lucene.util.InPlaceMergeSorter;
+
+// static utility functions for encoding and decoding dense_vector and sparse_vector fields
+public final class VectorEncoderDecoder {


can it be made pkg-private?

jpountz · 2018-12-06T16:43:19Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/VectorEncoderDecoder.java

+// static utility functions for encoding and decoding dense_vector and sparse_vector fields
+public final class VectorEncoderDecoder {
+    static final byte INT_BYTES = 4;
+    static final byte SHORT_BYTES = 2;


let's use Integer.BYTES and Short.BYTES instead

jpountz · 2018-12-06T16:46:38Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/VectorEncoderDecoder.java

+     * @param vectorBR - vector decoded in BytesRef
+     */
+    static int[] decodeSparseVectorDims(BytesRef vectorBR) {
+        int dimCount = (vectorBR.length - vectorBR.offset) / (INT_BYTES + SHORT_BYTES);


no need to subtract offset here, length is the number of bytes rather than the end offset

jpountz · 2018-12-06T16:48:13Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/VectorEncoderDecoder.java

+     * @param vectorBR - vector decoded in BytesRef
+     */
+    static float[] decodeSparseVector(BytesRef vectorBR) {
+        int dimCount = (vectorBR.length - vectorBR.offset) / (INT_BYTES + SHORT_BYTES);


no need to subtract vectorBR.offset

jpountz · 2018-12-06T16:53:29Z

...mapper-extras/src/test/java/org/elasticsearch/index/mapper/SparseVectorFieldMapperTests.java

+            0.001f
+        );
+    }
+}


let's test indexing illegal vectors and check error messages?

mayya-sharipova · 2018-12-07T19:27:28Z

@jpountz Thanks for the feedback, Adrien. I have addressed all your comments (except Integer.BYTES - which if necessary I can address as well).

jpountz

LGTM, thanks @mayya-sharipova for the iterations and sorry it took so long!

jpountz · 2018-12-12T14:11:58Z

modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/DenseVectorFieldMapper.java

+
+    public static final String CONTENT_TYPE = "dense_vector";
+    public static short MAX_DIMS_COUNT = 500; //maximum allowed number of dimensions
+    private static final byte INT_BYTES = 4;


mayya-sharipova · 2018-12-12T22:30:57Z

@elasticmachine run the gradle build tests 1

mayya-sharipova · 2018-12-13T02:21:15Z

@jpountz thanks a lot for the review

Relates to #33022

mayya-sharipova added 2 commits August 20, 2018 16:05

WIP vector_field

2acb191

mayya-sharipova added the >enhancement label Aug 21, 2018

mayya-sharipova added the :Search/Ranking Scoring, rescoring, rank evaluation. label Aug 21, 2018

mayya-sharipova requested a review from jpountz August 21, 2018 12:40

mayya-sharipova added the WIP label Aug 21, 2018

mayya-sharipova added 4 commits November 8, 2018 09:49

Merge remote-tracking branch 'upstream/master' into vector-field

3ebc80d

Introduce dense_vector and sparse_vector fields

df613db

Remove vector query Add documentation Add tests

afafad

ecce6df

Merge remote-tracking branch 'upstream/master' into vector-field

d4ef73d

mayya-sharipova removed the WIP label Nov 12, 2018

jpountz reviewed Nov 20, 2018

View reviewed changes

Merge branch 'master' into vector-field

9d7f7d0

mayya-sharipova force-pushed the vector-field branch from 41280f3 to 13986d2 Compare November 29, 2018 16:34

mayya-sharipova added 2 commits November 29, 2018 14:38

Address feedback 2

fea71a4

Merge remote-tracking branch 'upstream/master' into vector-field

bb57049

mayya-sharipova force-pushed the vector-field branch from 13986d2 to bb57049 Compare November 29, 2018 19:41

jtibshirani reviewed Nov 30, 2018

View reviewed changes

jpountz reviewed Nov 30, 2018

View reviewed changes

Address Julie's and Adrien's feedback

007b308

jpountz reviewed Dec 6, 2018

View reviewed changes

Address Adrien's comments

17212bd

jpountz mentioned this pull request Dec 12, 2018

DocValueFormat#format(BytesRef) to return String rather than Object #36504

Closed

jpountz approved these changes Dec 12, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master' into vector-field

8383f59

mayya-sharipova merged commit b5d532f into elastic:master Dec 13, 2018

mayya-sharipova deleted the vector-field branch December 13, 2018 02:21

mayya-sharipova added a commit that referenced this pull request Dec 13, 2018

Make vector fields experimental feature

bda0316

Relates to #33022

mayya-sharipova added the v7.0.0 label Dec 14, 2018

jpountz mentioned this pull request Dec 17, 2018

Rename feature_vector to feature_map? #36723

Closed

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

codebrain mentioned this pull request Jun 20, 2019

Add support for dense_vector and sparse_vector field mappings elastic/elasticsearch-net#3836

Closed

jtibshirani added :Search/Vectors Vector search and removed :Search/Ranking Scoring, rescoring, rank evaluation. labels Jul 21, 2022

Vector field #33022

Vector field #33022

Conversation

mayya-sharipova commented Aug 21, 2018 • edited

mayya-sharipova commented Aug 21, 2018

elasticmachine commented Aug 21, 2018

jpountz commented Aug 23, 2018

mayya-sharipova commented Nov 12, 2018

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Nov 29, 2018

jtibshirani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Nov 30, 2018 • edited

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Dec 7, 2018

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Dec 12, 2018

mayya-sharipova commented Dec 13, 2018

mayya-sharipova commented Aug 21, 2018 •

edited

mayya-sharipova commented Nov 30, 2018 •

edited