arsenal.maths package¶
Subpackages¶
Submodules¶
arsenal.maths.checkgrad module¶
-
arsenal.maths.checkgrad.fd_Jv(f, x, shape=None)[source]¶ Finite-difference approximation to a Jacobian-vector product, J[f](x) v, where f: Rⁿ ↦ Rᵐ.
Example: If f is a gradient, this function approximates Hessian-vector products.
Solve a linear system for second-order gradient methods without materializing the Hessian or Fisher matrix. For example, the natural gradient direction in an exponential family:
g = Euclidean_gradient(w) Fvp = fd_Jv(dlogZ, w) # Fisher-vector products ng = implicit_cg(Fvp, g)[0]
-
arsenal.maths.checkgrad.fdcheck(func, w, g, keys=None, eps=1e-05, quiet=0, verbose=1, progressbar=1, throw=True)[source]¶ Finite-difference check.
Returns arsenal.math.compare instance.
- func: zero argument function, which references w in caller’s scope.
- w: parameters.
- g: gradient estimate to compare against
- keys: dimensions to check
- eps: perturbation size
-
arsenal.maths.checkgrad.prox_numerical(f, x, s, jac=None)[source]¶ Numerically estimate the proximal operator Prox_{s*f}(x).
-
arsenal.maths.checkgrad.quick_fdcheck(func, w, g, n_checks=20, eps=1e-05, quiet=True, verbose=False, progressbar=False, throw=True)[source]¶ Check gradient along random directions (a faster alternative to axis-aligned directions).
Tim Vieira (2017) “How to test gradient implementations” https://timvieira.github.io/blog/post/2017/04/21/how-to-test-gradient-implementations/
arsenal.maths.cholesky module¶
Utilities for working with the Cholesky factorization of a positive-definite matrix.
-
class
arsenal.maths.cholesky.Cholesky(M)[source]¶ Bases:
objectInterface to Cholesky factorization of a matrix, which supports updates (growing and rank-one updates) as well as efficient utility methods (e.g., solve, det).
Remarks:
- We store the factorization (self.L) as upper triangular to be consistent with scipy.linalg.cholesky. Note that numpy.linalg.cholesky is lower triangular for some reason.
arsenal.maths.combinatorics module¶
arsenal.maths.compare module¶
-
arsenal.maths.compare.align(X, Y, distance=<function <lambda>>, maximize=False)[source]¶ Find a cheap alignment of X and Y.
arsenal.maths.featureselection module¶
Find most informative features ranked by information gain (i.e. the ID3 hueristic for decision trees).
Input: a tab-delimited file where each line starts with a label followed by features. Output: a sorted list of the most informative features.
-
class
arsenal.maths.featureselection.Alphabet(random_int=None)[source]¶ Bases:
objectBijective mapping from strings to integers.
>>> a = Alphabet() >>> [a[x] for x in 'abcd'] [0, 1, 2, 3] >>> list(map(a.lookup, range(4))) ['a', 'b', 'c', 'd']
>>> a.stop_growth() >>> a['e']
>>> a.freeze() >>> a.add('z') Traceback (most recent call last): ... ValueError: Alphabet is frozen. Key "z" not found.
>>> print(a.plaintext()) a b c d
-
add(k)¶
-
-
arsenal.maths.featureselection.integerize(data)[source]¶ Integerize dataset returns a triple (label alphabet, feature alphabet, integerized dataset)
-
arsenal.maths.featureselection.kl_divergence(p, q)[source]¶ Compute KL divergence of two vectors, K(p || q). NOTE: If any value in q is 0.0 then the KL-divergence is infinite.
-
arsenal.maths.featureselection.kl_filter(data, verbose=True, progress=False, out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>, feature_label_cuttoff=0, feature_count_cuttoff=0, do_label_count=False)[source]¶ data = (label, [features …])
KL is a synonym for Information Gain
KL( p(label) || p(label|feature) )
arsenal.maths.inv module¶
-
class
arsenal.maths.inv.InvAndDet(A)[source]¶ Bases:
arsenal.maths.inv.Inv
arsenal.maths.optimize module¶
arsenal.maths.pareto module¶
Pareto frontier
-
arsenal.maths.pareto.pareto_frontier(X, Y, maxX=True, maxY=True, indices=False)[source]¶ Determine Pareto frontier, returns list of sorted points.
Args:
X, Y: data.
- maxX, maxY: (bool) whether to maximize or minimize along respective
- coordinate.
-
arsenal.maths.pareto.pareto_ix(X, Y, *a, **kw)[source]¶ Determine Pareto frontier, returns list of indices
-
arsenal.maths.pareto.show_frontier(X, Y, maxX=False, maxY=True, dots=False, XMIN=None, XMAX=None, YMIN=None, YMAX=None, ax=None, label=None, interpolation='pessimistic', **style)[source]¶ Plot Pareto frontier.
Args:
X, Y: data.
- maxX, maxY: (bool) whether to maximize or minimize along respective
- coordinate.
- dots: (bool) highlight points on the frontier (will use same color as
- style).
ax: use an existing axis if non-null.
- style: keyword arguments, which will be passed to lines connecting the
- points on the Pareto frontier.
XMAX: max value along x-axis YMIN: min value along y-axis
arsenal.maths.rvs module¶
-
class
arsenal.maths.rvs.Empirical(x)[source]¶ Bases:
objectEmpirical CDF of data a, returns function which makes values to their cumulative probabilities.
>>> g = cdf([5, 10, 15])
Evaluate the CDF at a few points
>>> g([5,9,13,15,100]) array([0.33333333, 0.33333333, 0.66666667, 1. , 1. ])
Check that ties are handled correctly
>>> g = cdf([5, 5, 15])
The value p(x <= 5) = 2/3
>>> g([0, 5, 15]) array([0. , 0.66666667, 1. ])
The auantile function should be the inverse of the cdf.
>>> g = cdf([-1, 5, 5, 15]) >>> g.quantile(np.linspace(0, 1, 10)) array([-1, -1, -1, 5, 5, 5, 5, 5, 5, 15])
-
cdf(z)¶ Call self as a function.
-
ppf(q)¶
-
-
arsenal.maths.rvs.cdf¶ alias of
arsenal.maths.rvs.Empirical
-
arsenal.maths.rvs.random_dist(*size)[source]¶ Generate a random conditional distribution which sums to one over the last dimension of the input dimensions.
arsenal.maths.stepsize module¶
Adaptive stepsize algorithms
-
class
arsenal.maths.stepsize.adam(x)[source]¶ Bases:
objectAdam as described in http://arxiv.org/pdf/1412.6980.pdf.
arsenal.maths.util module¶
-
arsenal.maths.util.assert_equal(a, b, name='', verbose=False, throw=True, tol=0.001, color=1)[source]¶ isfinite: asserts that both a and b must be finite.
>>> assert_equal(0, 1, throw=0, color=0) 0 1 err=[1.] fail
>>> assert_equal(0, 0, verbose=1, color=0) 0 0 err=[0] ok
-
arsenal.maths.util.bernstein(samples, delta, R, check=True)[source]¶ Plug-n-chug empirical Bernstein bound, computes “error bars” which hold with probability 1-delta for the mean of independent samples from a given range R=b-a (known a priori).
Returns epsilon such that the following bound hold,
p( mean(samples) - true_mean <= eps ) >= 1-deltaWe assume that sample are independent (not necessarily identically distributed).
Bound is based on sample variance V and a priori knowledge that RVs in are in the range [a,b] (although we only really require a known range b-a)
The bound holds with probability >=(1-delta).
The sample mean has symmetric deviations so we get a two-sided bound by passing in 2*delta, i.e.,
p( |mean(samples) - true_mean| <= eps ) >= 1-2*deltaThis is analogous to p-values, which make assumption of normally distributed random variables. This means that the bounds can be ‘tighter’, but the assumptions are usually not valid.
-
arsenal.maths.util.blocks(M, k)[source]¶ Partition the matrix M into blocks
- M = [[A, B],
- [C, D]]
Output: A: k x k, D: (n-k) x (n-k) if M: n x n.
TODO: support arbitrary partitions when k is a list
-
arsenal.maths.util.cross_entropy(p, q)[source]¶ Cross Entropy of two vectors,
CE(p,q) = - sum_i p[i] log q[i]
Relationship to KL-divergence:
CE(p,q) = entropy(p) + KL(p||q)
-
arsenal.maths.util.cumavg(x)[source]¶ Cumulative average.
>>> cumavg([1,2,3,4,5]) array([1. , 1.5, 2. , 2.5, 3. ])
-
arsenal.maths.util.d_softmax(out, x, adj)[source]¶ out = softmax(x), adj are the adjoints we are chaining together.
A(x) = logsumexp(x)∇ A(x) = softmax(x) ∇² A(x) = d_softmax(x) % this method.
-
arsenal.maths.util.f1(A, B)[source]¶ Compute the F1 measure on two bit vectors.
>>> f1([1], [1]) 1.0
>>> f1([1], [0]) 0.0
>>> f1([1,0,1], [0,1,1]) 0.5
-
arsenal.maths.util.kl_divergence(p, q)[source]¶ Compute KL divergence of two vectors, K(p || q). NOTE: If any value in q is 0.0 then the KL-divergence is infinite.
-
arsenal.maths.util.lidstone(p, delta)[source]¶ Lidstone smoothing is a generalization of Laplace smoothing.
-
arsenal.maths.util.log1mexp(x)[source]¶ Numerically stable implementation of log(1-exp(x))
Note: function is finite for x < 0.
Source: http://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
-
arsenal.maths.util.logsubexp(x, y)[source]¶ Numerically stable computation of subtraction in log-space z = log(exp(x) - exp(y))
-
arsenal.maths.util.logsumexp(arr, axis=None)[source]¶ Computes the sum of arr assuming arr is in the log domain.
Returns log(sum(exp(arr))) while minimizing the possibility of over/underflow.
Examples:
>>> a = arange(10) >>> log(sum(exp(a))) 9.45862974442671
>>> logsumexp(a) 9.45862974442671
>>> x = [[0, 0, 1000.0], [1000.0, 0, 0]] >>> logsumexp(x, axis=1) array([1000., 1000.])
>>> logsumexp(x) 1000.6931471805599
>>> logsumexp(x, axis=0) array([1.00000000e+03, 6.93147181e-01, 1.00000000e+03])
-
arsenal.maths.util.mean_confidence_interval(a, confidence=0.95)[source]¶ returns (mean, lower, upper)
-
arsenal.maths.util.mutual_information(joint)[source]¶ Mutual Information
MI(x,y) = KL( p(x,y) || p(x) p(y) )
We can compute this easily from the joint distribution
joint = p(X=x,Y=y)- because
- p(X=x) = sum_y p(X=x, Y=y) p(Y=y) = sum_x p(X=x, Y=y)
- relationships:
- MI(x,y) is the expected PMI(x,y) wrt p(x,y) MI(x,y) = KL(p(x,y) || p(x) p(y))
- properties:
- MI(X,Y) = MI(Y,X) is symmetric
-
arsenal.maths.util.normalize_interval(data)[source]¶ Shift and rescale data so that it lies in the range [0,1].
-
arsenal.maths.util.normalize_zscore(data)[source]¶ Shift and rescale data to be zero-mean and unit-variance along axis 0.
-
arsenal.maths.util.onehot(i, n)[source]¶ Create a one-hot vector: a vector of length n with a 1 at position i and zeros elsewhere.
-
arsenal.maths.util.project_onto_simplex(a, radius=1.0)[source]¶ Project point a to the probability simplex. Returns (the projected point x, the zero threshold, and the residual value).
-
arsenal.maths.util.relative_difference(a, b)[source]¶ Element-wise relative difference of two arrays
Choices for denominator include:
- Further reading:
- http://en.wikipedia.org/wiki/Relative_change_and_difference
-
arsenal.maths.util.set_printoptions(*args, **kw)[source]¶ Context manager for numpy’s print options.
-
arsenal.maths.util.simpless(w, B)[source]¶ - Reparameterization transform similar to softmax, but for the constraints set
- {p | p >= 0, sum(p) <= B}.
-
arsenal.maths.util.softmax(x, axis=None)[source]¶ >>> x = [1, -10, 100, .5] >>> softmax(x) array([1.01122149e-43, 1.68891188e-48, 1.00000000e+00, 6.13336839e-44])
>>> exp(x) / exp(x).sum() array([1.01122149e-43, 1.68891188e-48, 1.00000000e+00, 6.13336839e-44])
>>> x = [[0, 0, 1000], [1000, 0, 0]]
Normalize by row: >>> softmax(x, axis=0) array([[0. , 0.5, 1. ],
[1. , 0.5, 0. ]])Normalize by column: >>> softmax(x, axis=1) array([[0., 0., 1.],
[1., 0., 0.]])Normalize by cell: >>> softmax(x, axis=None) array([[0. , 0. , 0.5],
[0.5, 0. , 0. ]])
-
arsenal.maths.util.split_ix(N, p, randomize=1)[source]¶ Sample a random partition of N integers with proportions p.