Together with my supervisor Philipp Hennig, I am working on probabilistic numerical methods to solve large linear equation systems.
We believe that numerical algorithms are actually Bayesian inference algorithms:
These algorithms gather data and propose an estimate based on these observations.
Different algorithms usually produce different estimates even if given the same data (e.g. FOM and GMRES) as they usually have different underlying assumptions.
One of our goals is to make these assumptions explicit using the language of probability in form of a prior belief over the solution.
In addition to residual or worst-case error, such a view provides an uncertainty estimate regarding the quality of the approximation.
This uncertainty allows to reason over the average error and it can be propagated to the application that called for the solution.
Furthermore, a Bayesian view allows (at least in theory) to incorporate additional prior knowledge leading to more specifically tailored algorithms for specific applications.
A particular application we are interested in are linear equation systems in which the coefficient matrix stems from a distribution.
For example in Gaussian process regression it is common to assume that the underlying dataset is independently and identically distributed.
This implies that the rows of the kernel matrix are correlated, i.e. a structure that we aim to use to solve such linear equation system faster.
Artificial intelligent systems build models of their environment from observations, and choose actions that they predict will have beneficial effect on the environment's state. A central challenge is that the mathematical models used in this process call for computations that have no closed, analytic solution. Learning machines ...
Bayesian Optimization is an increasingly popular approach to industrial and scientific prototyping problems. The basic premise in this setting is that one is looking for a location $x$ in some domain where a fitness function $f(x)$ is (globally) minimized. The additional, sometimes implicit, assumption is t...
Linear algebra methods form the basis for the majority of numerical computations. Because they perform a positively elementary task, they have to satisfy strict requirements on computational efficiency and numerical robustness.
Our work has added to a growing understanding that many, widely used, linear solvers c...
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017), 52, JMLR Workshop and Conference Proceedings, (Editors: Sign, Aarti and Zhu, Jerry), 2017 (conference) Accepted
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016), 51, pages: 676-684, JMLR Workshop and Conference Proceedings, (Editors: Gretton, A. and Robert, C. C. ), 2016 (conference)
Least-squares and kernel-ridge / Gaussian process regression are among the foundational algorithms of statistics and machine learning. Famously, the worst-case cost of exact nonparametric regression grows cubically with the data-set size; but a growing number of approximations have been developed that estimate good solutions at lower cost. These algorithms typically return point estimators, without measures of uncertainty. Leveraging recent results casting elementary linear algebra operations as probabilistic inference, we propose a new approximate method for nonparametric least-squares that affords a probabilistic uncertainty estimate over the error between the approximate and exact least-squares solution (this is not the same as the posterior variance of the associated Gaussian process regressor). This allows estimating the error of the least-squares solution on a subset of the data relative to the full-data solution. The uncertainty can be used to control the computational effort invested in the approximation. Our algorithm has linear cost in the data-set size, and a simple formal form, so that it can be implemented with a few lines of code in programming languages with linear algebra functionality.
Bartels, S., Hennig, P.
Probabilistic Approximate Least-SquaresProceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016), 51, pages: 676-684, JMLR Workshop and Conference Proceedings, (Editors: Gretton, A. and Robert, C. C. ), 2016 (conference)
Our goal is to understand the principles of Perception, Action and Learning in autonomous systems that successfully interact with complex environments and to use this understanding to design future systems