{"abstracts":[{"sha1":"16377014e7a3233e8ee0e122e861145d9825fa7b","content":"The Rashomon effect occurs when many different explanations exist for the\nsame phenomenon. In machine learning, Leo Breiman used this term to\ncharacterize problems where many accurate-but-different models exist to\ndescribe the same data. In this work, we study how the Rashomon effect can be\nuseful for understanding the relationship between training and test\nperformance, and the possibility that simple-yet-accurate models exist for many\nproblems. We consider the Rashomon set - the set of almost-equally-accurate\nmodels for a given problem - and study its properties and the types of models\nit could contain. We present the Rashomon ratio as a new measure related to\nsimplicity of model classes, which is the ratio of the volume of the set of\naccurate models to the volume of the hypothesis space; the Rashomon ratio is\ndifferent from standard complexity measures from statistical learning theory.\nFor a hierarchy of hypothesis spaces, the Rashomon ratio can help modelers to\nnavigate the trade-off between simplicity and accuracy. In particular, we find\nempirically that a plot of empirical risk vs. Rashomon ratio forms a\ncharacteristic Γ-shaped Rashomon curve, whose elbow seems to be a\nreliable model selection criterion. When the Rashomon set is large, models that\nare accurate - but that also have various other useful properties - can often\nbe obtained. These models might obey various constraints such as\ninterpretability, fairness, or monotonicity.","mimetype":"text/plain","lang":"en"},{"sha1":"523494747c410111fe56405313c4483ddb5b3c14","content":"The Rashomon effect occurs when many different explanations exist for the\nsame phenomenon. In machine learning, Leo Breiman used this term to\ncharacterize problems where many accurate-but-different models exist to\ndescribe the same data. In this work, we study how the Rashomon effect can be\nuseful for understanding the relationship between training and test\nperformance, and the possibility that simple-yet-accurate models exist for many\nproblems. We consider the Rashomon set - the set of almost-equally-accurate\nmodels for a given problem - and study its properties and the types of models\nit could contain. We present the Rashomon ratio as a new measure related to\nsimplicity of model classes, which is the ratio of the volume of the set of\naccurate models to the volume of the hypothesis space; the Rashomon ratio is\ndifferent from standard complexity measures from statistical learning theory.\nFor a hierarchy of hypothesis spaces, the Rashomon ratio can help modelers to\nnavigate the trade-off between simplicity and accuracy. In particular, we find\nempirically that a plot of empirical risk vs. Rashomon ratio forms a\ncharacteristic $\\Gamma$-shaped Rashomon curve, whose elbow seems to be a\nreliable model selection criterion. When the Rashomon set is large, models that\nare accurate - but that also have various other useful properties - can often\nbe obtained. These models might obey various constraints such as\ninterpretability, fairness, or monotonicity.","mimetype":"application/x-latex","lang":"en"}],"refs":[],"contribs":[{"index":0,"raw_name":"Lesia Semenova","role":"author"},{"index":1,"raw_name":"Cynthia Rudin","role":"author"},{"index":2,"raw_name":"Ronald Parr","role":"author"}],"license_slug":"ARXIV-1.0","language":"en","version":"v2","ext_ids":{"arxiv":"1908.01755v2"},"release_year":2020,"release_date":"2020-03-27","release_stage":"submitted","release_type":"article","webcaptures":[],"filesets":[],"files":[{"release_ids":["lzafipvlvvefxc6oqbosov347m"],"mimetype":"application/pdf","urls":[{"url":"https://arxiv.org/pdf/1908.01755v2.pdf","rel":"repository"},{"url":"https://web.archive.org/web/20200331000820/https://arxiv.org/pdf/1908.01755v2.pdf","rel":"webarchive"}],"sha256":"8f8c5f34ace74f18aa2a916463d30d41248f16f52e01b34f8019b79710a2cdd3","sha1":"51df389c5be8e16c1e8896803f290a082ad53d60","md5":"80bdd86b2d4e0970a481b7b2aee2b4c2","size":8716656,"revision":"afb72c14-4c9f-4072-b71b-27b3368cee89","ident":"zczytkoxxvhhpiobhrei7qauve","state":"active"}],"work_id":"4jpekfxphvdwleenhbfgbc6nfm","title":"A study in Rashomon curves and volumes: A new perspective on\n generalization and model simplicity in machine learning","state":"active","ident":"lzafipvlvvefxc6oqbosov347m","revision":"c7c518a5-3292-4106-98a5-cdc801cc9c45","extra":{"arxiv":{"base_id":"1908.01755","categories":["cs.LG","stat.ML"],"comments":"Revisited sections 4, 7, and 8"}}}