Variational Inference in Reduced Order Dynamical Models (2009)
Cornford, Dan; NCRG, Aston University; D.cornford@aston.ac.uk, Shen, Yuan; NCRG, Aston University; Y.shen2@aston.ac.uk, Opper, Manfred; Artificial Intelligence Group, TU Berlin; Opperm@cs.tu-berlin.de
Dynamical systems arise across a range of application domains, from systems biology, to weather forecasting. To study such systems it is necessary to build models to represent the important processes...
Efficient statistical inference for stochastic reaction processes (2009)
Ruttor, Andreas, Opper, Manfred
We address the problem of estimating unknown model parameters and state variables in stochastic reaction processes when only sparse and noisy measurements are available. Using an asymptotic system...
The Variational Gaussian Approximation Revisited (2009)
Opper, Manfred, Archambeau, Cedric
The variational approximation of posterior distributions by multivariate Gaussians has been much less popular in the Machine Learning community compared to the corresponding approximation by...
Approximate inference for stochastic reaction processes. (2009)
Ruttor, Andreas, Sanguinetti, Guido, Opper, Manfred
We discuss the problem of statistical inference for Markov jump processes modelling biochemical reactions. Using a variational formulation of exact inference we derive two different approximations. A...
Shen, Yuan, Archambeau, Cedric, Cornford, Dan, Opper, Manfred, Shawe-Taylor, John, Barillec, Remi
In recent years we have developed a novel variational method for partially observed systems governed by stochastic differential equations. In this paper we provide a comparison of the Variational...
The Variational Gaussian Approximation Revisited. (2009)
Opper, Manfred, Archambeau, Cedric
The variational approximation of posterior distributions by multivariate Gaussians has been much less popular in the Machine Learning community compared to the corresponding approximation by...
Switching Regulatory Models of Cellular Stress Response. (2009)
Sanguinetti, Guido, Ruttor, Andreas, Opper, Manfred
\section{Motivation}Stress response in cells is often mediated by quick activation of transcription factors. Given the difficulty in experimentally assaying transcription factor activities, several...
Switching regulatory models of cellular stress response (2009)
Sanguinetti, Guido, Ruttor, Andreas, Opper, Manfred, Archambeau, Cedric
Motivation: Stress response in cells is often mediated by quick activation of transcription factors (TFs). Given the difficulty in experimentally assaying TF activities, several statistical...
Switching Regulatory Models of Cellular Stress Response (2008)
Sanguinetti, Guido, Ruttor, Andreas, Opper, Manfred, Archambeau, Cedric
Stress response in cells is often mediated by quick activation of transcription factors. Given the difficulty in experimentally assaying transcription factor activities, several statistical...
Variational inference for Markov jump processes (2008)
Opper, Manfred, Sanguinetti, Guido
Discrete stochastic processes play an important role in a large number of application domains. However, realistic systems are analytically intractable and they have traditionally been analysed using...
Shen, Yuan, Archambeau, Cedric, Cornford, Dan, Opper, Manfred
In this paper, we develop a set of novel Markov chain Monte Carlo algorithms for Bayesian inference in partially observed non-linear diffusion processes. The Markov chain Monte Carlo algorithms we...
Improving on Expectation Propagation. (2008)
Opper, Manfred, Paquet, Ulrich, Winther, Ole
A series of corrections is developed for the fixed points of Expectation Propagation (EP), which is one of the most popular methods for approximate probabilistic inference. These corrections can lead...
Variational Inference for Diffusion Processes (2007)
Archambeau, Cedric, Opper, Manfred, Shen, Yuan, Cornford, Dan, Shawe-Taylor, John
Diffusion processes are a family of continuous-time continuous-state stochastic processes that are in general only partially observed. The joint estimation of the forcing parameters and the system...
Variational Inference for Markov Jump Processes (2007)
Opper, Manfred, Sanguinetti, Guido
Markov jump processes play an important role in a large number of application domains. However, realistic systems are analytically intractable and they have traditionally been analysed using...
A new method to calculate the full training process of a neural network is introduced. No sophisticated methods like the replica trick are used. The results are directly related to the actual number...
Shen, Yuan, Archambeau, Cedric, Cornford, Dan, Opper, Manfred, Shawe-Taylor, John, Barillec, Remi
In recent work we have developed a novel variational inference method for partially observed systems governed by stochastic differential equations. In this paper we provide a comparison of the...
Shen, Yuan, Archambeau, Cedric, Cornford, Dan, Opper, Manfred, Shawe-Taylor, John, Barillec, Remi
In recent work we have developed a novel variational inference method for partially observed systems governed by stochastic differential equations. In this paper we provide a comparison of the...
Gaussian Process Approximations of Stochastic Differential Equations (2007)
Archambeau, Cedric, Cornford, Dan, Opper, Manfred, Shawe-Taylor, John
Stochastic differential equations arise naturally in a range of contexts, from financial to environmental modeling. Current solution methods are limited in their representation of the posterior...
Gaussian Process Approximations of Stochastic Differential Equation (2007)
Archambeau, Cedric, Cornford, Dan, Opper, Manfred, Shawe-Taylor, John
Stochastic differential equations arise naturally in a range of contexts, from financial to environmental modeling. Current solution methods are limited in their representation of the posterior...
Variational Inference for Diffusion Processes (2007)
Archambeau, Cedric, Opper, Manfred, Shen, Yuan, Cornford, Dan, Shawe-Taylor, John
Diffusion processes are a family of continuous-time continuous-state stochastic processes that are in general only partly observed. The joint estimation of the forcing parameters and the system noise...
Gaussian Process Approximations of Stochastic Differential Equations (2006)
Archambeau, Cedric, Cornford, Dan, Opper, Manfred, Shawe-Taylor, John
Stochastic differential equations arise naturally in a range of contexts, from financial to environmental modelling. Current solution methods are based on a range of strong and weak approximation...
Expectation Consistent Approximate Inference (2005)
We propose a novel framework for approximations to intractable probabilistic models which is based on a free energy formulation. The approximation can be understood from replacing an average over the...
Approximate inference techniques with expectation constraints (2005)
Heskes, Tom, Opper, Manfred, Wiegerinck, Wim, Winther, Ole, Zoeter, Onno
This paper discusses inference problems in probabilistic graphical models that often occur in a machine learning setting. In particular it presents a unified view of several recently proposed...
Approximate inference techniques with expectation constraints (2005)
Heskes, Tom, Opper, Manfred, Wiegerinck, Wim, Winther, Ole, Zoeter, Onno
This article discusses inference problems in probabilistic graphical models that often occur in a machine learning setting. In particular it presents a unified view of several recently proposed...
An approximate inference approach for the pca reconstruction error (2005)
The problem of computing a resample estimate for the reconstruction error in PCA is reformulated as an inference problem with the help of the replica method. Using the expectation consistent (EC)...
Expectation Consistent Free Energies for Approximate Inference (2005)
We propose a novel a framework for deriving approximations for intractable probabilistic models. This framework is based on a free energy (negative log marginal likelihood) and can be seen as a...
A statistical physics approach for the analysis of machine learning algorithms on real data. (2005)
Opper, Manfred, Opper, Manfred
We combine the replica approach of statistical physics with a variational technique to make it applicable for the analysis of machine learning algorithms on real data. The method is applied to...
Approximate Inference in Probabilistic Models (2004)
We present a framework for approximate inference in probabilistic data models which is based on free energies. The free energy is constructed from two approximating distributions which encode...
Bayesian Analysis of the Scatterometer Wind Retrieval inverse Problem: Some new Approaches (2004)
Cornford, Dan, Csato, Lehel, Evans, David J, Opper, Manfred
The retrieval of wind vectors from satellite observed radar backscatter can be seen as a non-linear inverse problem. A common approach to solving inverse problems is the Bayesian framework: to infer...
Expectation Consistent Approximate Inference (2004)
We propose a novel framework for approximations to intractable probabilistic models. The method is based on a free energy formulation of inference and allows for a simultaneous computation of...
An Approximate Analytical Approach to Resampling Averages (2003)
Malzahn, Doerthe, Opper, Manfred
Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our...
An Approximate Analytical Approach to Resampling Averages (2003)
Malzahn, Doerthe, Opper, Manfred
Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our...
An Approximate Analytical Approach to Resampling Averages (2003)
Malzahn, Doerthe, Opper, Manfred
Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our...
An Approximate Analytical Approach to Resampling Averages (2003)
Malzahn, Doerthe, Opper, Manfred
Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our...
An Approximate Analytical Approach to Resampling Averages (2003)
Malzahn, Doerthe, Opper, Manfred
Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our...
An Approximate Analytical Approach to Resampling Averages (2003)
Malzahn, Doerthe, Opper, Manfred
Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our...
Variational Linear Response (2003)
A general linear response method for deriving improved estimates of correlations in the variational Bayes framework is presented. Three applications are given and it is discussed how to use linear...
Approximate Analytical Bootstrap Averages for Support Vector Classifiers (2003)
We compute approximate analytical bootstrap averages for support vector classification using a combination of the replica method of statistical physics and the TAP approach for approximate inference....
Sparse Online Gaussian Processes (2002)
We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations caused by large data sets. The...
Sparse Online Gaussian Processes (2002)
We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations caused by large data sets. The...
Sparse Online Gaussian Processes (2002)
We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations caused by large data sets. The...
Sparse Online Gaussian Processes (2002)
We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations caused by large data sets. The...
Sparse Online Gaussian Processes (2002)
We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations caused by large data sets. The...
Sparse Online Gaussian Processes (2002)
We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations caused by large data sets. The...
Tractable Inference for Probabilistic Data Models (2002)
Lehel Csato, Manfred Opper, Ole Winther
We present an approximation technique for probabilistic data models with a large number of hidden variables, based on ideas from statistical physics. We give examples for two nontrivial applications....
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Universal learning curves of support vector machines (2001)
Opper, Manfred, Urbanczik, Robert
Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Universal learning curves of support vector machines (2001)
Opper, Manfred, Urbanczik, Robert
Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Universal learning curves of support vector machines (2001)
Opper, Manfred, Urbanczik, Robert
Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Universal learning curves of support vector machines (2001)
Opper, Manfred, Urbanczik, Robert
Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Universal learning curves of support vector machines (2001)
Opper, Manfred, Urbanczik, Robert
Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Tractable approximations for probabilistic models: The adaptive TAP mean field approach (2001)
We develop an advanced mean field method for approximating averages in probabilistic data models that is based on the TAP approach of disorder physics. In contrast to conventional TAP, where the...
Universal learning curves of support vector machines (2001)
Opper, Manfred, Urbanczik, Robert
Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the...
Sparse representation for Gaussian process models (2001)
We develop an approach for a sparse representation for Gaussian Process (GP) models in order to overcome the limitations of GPs caused by large data sets. The method is based on a combination of a...
Sparse representation for Gaussian process models (2001)
We develop an approach for a sparse representation for Gaussian Process (GP) models in order to overcome the limitations of GPs caused by large data sets. The method is based on a combination of a...
From Naive Mean Field Theory to the TAP Equations (2001)
Manfred Opper, Ole Winther, London England
Introduction Mean field (MF) methods provide tractable approximations for the computation of high dimensional sums and integrals in probabilistic models. By neglecting certain dependencies between...
Efficient Approaches to Gaussian Process Classification (2000)
Lehel Csato, Ernest Fokoué, Manfred Opper, Bernhard Schottky, Ole Winther
We present three simple approximations for the calculation of the posterior mean in Gaussian Process classification. The first two methods are related to mean field ideas known in Statistical...
Efficient Approaches to Gaussian Process Classification (2000)
Lehel Csato, Ernest Fokoué, Manfred Opper, Bernhard Schottky, Ole Winther
We present three simple approximations for the calculation of the posterior mean in Gaussian Process classification. The first two methods are related to mean field ideas known in Statistical...
Introduction Mean field (MF) methods provide efficient approximations which are able to cope with the increasing complexity of modern probabilistic data models. They replace the intractable task of...
Continuous Drifting Games (2000)
We combine the results of [5] and [3] and derive a continuous variant of a large class of drifting games. Our analysis furthers the understanding of the relationship between boosting, drifting games...
Efficient Approaches to Gaussian Process Classification (2000)
Lehel Csato, Ernest Fokoue, Manfred Opper, Bernhard Schottky, Ole Winther
.71> y = sign (ha(x)i) where ha(x)i is the posterior mean: ha(x)i = E a(x) Q t i=1 P (y i ja(x i )) E Q t i=1 P (y i ja(x i )) E is the expectation over the GP prior and t is the number of...
Gaussian Processes for Classification: Mean Field Algorithms (1999)
We derive a mean field algorithm for binary classification with Gaussian processes which are based on the TAP approach originally proposed in statistical physics of disordered systems. The theory...
Mean field methods for classification with Gaussian processes (1999)
We discuss the application of TAP mean field methods known from the Statistical Mechanics of disordered systems to Bayesian classification models with Gaussian processes. In contrast to previous...
Statistical Mechanics of Support Vector Networks (1998)
Dietrich, Rainer, Opper, Manfred, Sompolinsky, Haim
Using methods of Statistical Physics, we investigate the generalization performance of support vector machines (SVMs), which have been recently introduced as a general alternative to neural networks....
Statistical Mechanics of Learning in the Presence of Outliers (1998)
Dietrich, Rainer, Opper, Manfred
Using methods of statistical mechanics, we analyse the effect of outliers on the supervised learning of a classification problem. The learning strategy aims at selecting informative examples and...
Mutual information, metric entropy and cumulative relative entropy risk (1997)
Haussler, David, Opper, Manfred
Assume ${P_{\theta}: \theta \epsilon \Theta}$ is a set of probability distributions with a common dominating measure on a complete separable metric space Y. A state $\theta^* \epsilon \Theta$ is...
Regression with Gaussian Processes: Average Case Performance (1997)
. Recently, new models for regression and classification have been introduced which may be interpreted as neural networks in the limit of infinitely many parameters. For a regression model, the...
Metric Entropy and Minimax Risk in Classification (1997)
. We apply recent results on the minimax risk in density estimation to the related problem of pattern classification. The notion of loss we seek to minimize is an information theoretic measure of how...
Worst Case Prediction over Sequences under Log Loss (1997)
. We consider the game of sequentially assigning probabilities to future data based on past observations under logarithmic loss. We are not making probabilistic assumptions about the generation of...
Mutual information, metric entropy and cumulative relative entropy risk (1997)
is a set of probability distributions with a common dominating measure on a complete separable metric space Y. A state � * � � is chosen by Nature. A statistician obtains n independent...
Mutual Information, Metric Entropy, and Cumulative Relative Entropy Risk (1996)
Assume fP ` : ` 2 \Thetag is a set of probability distributions with a common dominating measure on a complete separable metric space Y . A state ` 2 \Theta is chosen by Nature. A statistician gets n...
Mutual Information, Metric Entropy, and Risk in Estimation of Probability Distributions (1996)
Assume fP ` : ` 2 \Thetag is a set of probability distributions with a common dominating measure on a complete separable metric space Y . A state ` 2 \Theta is chosen by Nature. A statistician gets n...
The Dynamics of Training Including Weight Decay (1996)
Siegfried Bös, Siegfried B, Manfred Opper
A new method to calculate the full training process of a neural network is introduced. It requires no such sophisticated methods as the replica trick. The results are directly related to the actual...
Selection of Examples for a Linear Classifier (1995)
Georg Jung, Manfred Opper, Georg Jungyand
. We investigate the problem of selecting an informative subsample out of a neural network's training data. Using the replica method of statistical mechanics, we calculate the performance of a...
Bounds for Predictive Errors in the Statistical Mechanics of Supervised Learning (1995)
Within a Bayesian framework, by generalizing inequalities known from statistical mechanics, we calculate general upper and lower bounds for a cumulative entropic error, which measures the success in...
David Haussler, Manfred Opper, Y Yn
Each parameter ` in an abstract parameter space \Theta is associated with a different probability distribution on a set Y . A parameter ` is chosen at random from \Theta according to some a priori...
Bounds for Predictive Errors in the Statistical Mechanics of Supervised Learning (1995)
Within a Bayesian framework, by generalizing inequalities known from statistical mechanics, we calculate general upper and lower bounds for a cumulative entropic error, which measures the success in...
Bounds for Predictive Errors in the Statistical Mechanics of Supervised Learning (1995)
Within a Bayesian framework, by generalizing inequalities known from statistical mechanics, we calculate general upper and lower bounds for a cumulative entropic error, which measures the success in...
Perceptron Learning: The Largest Version Space (1995)
Michael Biehl, Michael Biehl, Manfred Opper
We revisit the learning of a linearly separable rule with a single layer perceptron. The rule is taken to be correlated with a set of random training inputs, such that the concept is located in the...
General bounds for predictive errors in supervised learning (1995)
Within a Bayesian framework, we calculate general upper and lower bounds for a cumulative entropic error, which measures the success in the supervised learning of an unknown rule from examples. This...
Mutual information and Bayes methods for learning a distribution (1995)
Each parameter w in an abstract parameter space W is associated with a di erent probability distribution on a set Y. A parameter w is chosen at random from W according to some a priori distribution...
The learning curve of Bayes optimal classification algorithm when learning a perceptron from noisy random training examples is calculated exactly in the limit of large training sample size and large...
Bayesian analysis of the scatterometer wind retrieval inverse problem: some new approaches
Dan Cornford, Lehel Csató, David J. Evans, Manfred Opper
The retrieval of wind vectors from satellite scatterometer observations is a non-linear inverse problem. A common approach to solving inverse problems is to adopt a Bayesian framework and to infer...
Gaussian Process Classification and SVM: Mean Field Results and Leave-One-Out Estimator
In this chapter, we elaborate on the well-known relationship between Gaussian Processes (GP) and Support Vector Machines (SVM). Secondly, we present approximate solutions for two computational...