Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Learning rate #106
Learning rate #106
Conversation
Fixed dependencies in pom.xml
…hod. This allows the NAME to be used elsewhere instead of hardcoding the string.
…coding the string. added methods isFloating(), isInteger(), isNUmeric(), isBoolean() and isString()
…ng OP_NAME" to each generated operation.
…EntropyWitLogits() Added tf.nn.sparesSoftmaxCrossEntropyWithLogits() and tf.nn.raw.sparesSoftmaxCrossEntropyWithLogits() Added tf.nn.sigmoidCrossEntropyWithLogits()
…Logits to org.tensorflow.op.nn.raw
fix javadoc, fix casts
…yWithLogits.java to nn.raw, added new versions of these to NnOps
…maxCrossEntropyWithLogits()
…maxCrossEntropyWithLogits()
…lder into each Optimizer. Also, added to each Optimizer a corresponding Tensor that holds the value of the learning rate, and added a feed dictionary that maps the placeholder to the Tensor, so that it can be fed into the runner when running or evaluating. When setLearning rate is called the learning rate tensor and the feed dictionary are updated.
…lder into each Optimizer. Also, added to each Optimizer a corresponding Tensor that holds the value of the learning rate, and added a feed dictionary that maps the placeholder to the Tensor, so that it can be fed into the runner when running or evaluating. When setLearning rate is called the learning rate tensor and the feed dictionary are updated.
…x JavaDoc. Change from snake case to camel case.
…x JavaDoc. Change from snake case to camel case.
…Java files for some Ops. This also resulted in new generated source that are also committed.
…gmoidCrossEntropyWithLogits.java, and SparseSoftmaxCrossEntropyWithLogits.java under package org.tensorflow.op.nn in
…asy inclusion of a default optimizer. Cleaned up JavaDoc
…cases to framework. Created Tests for GradientDescent and Momentum
|
A couple of quick observations as I start reading this code. |
| } | ||
|
|
||
| /** Returns true if this data type represents a floating point type */ | ||
| public boolean isFloating() { |
deansher
Sep 17, 2020
Contributor
This pattern is very uncomfortable to me: DataType being omniscient about TType and dispatching on a string NAME. What's our motivation? If we think it's the best pattern for this situation, perhaps we could document why?
This pattern is very uncomfortable to me: DataType being omniscient about TType and dispatching on a string NAME. What's our motivation? If we think it's the best pattern for this situation, perhaps we could document why?
deansher
Sep 19, 2020
Contributor
Never mind, in this context -- I see in my local diff that this delta is unrelated to this PR. I'll raise this as an issue.
Never mind, in this context -- I see in my local diff that this delta is unrelated to this PR. I'll raise this as an issue.
| * @param graph the TensorFlow Graph | ||
| * @param name the name for this Optimizer (defaults to 'Adadelta') | ||
| * @param learningRate the learning rate | ||
| */ | ||
| public AdaDelta(Graph graph, String name, float learningRate) { | ||
| this(graph, name, learningRate, 0.95f, 1e-8f); |
deansher
Sep 17, 2020
Contributor
-> RHO_DEFAULT, EPSILON_DEFAULT
-> RHO_DEFAULT, EPSILON_DEFAULT
deansher
Sep 19, 2020
Contributor
Never mind, in this context.
Never mind, in this context.
Sync Clarke fork
| */ | ||
| protected Optimizer(Graph graph, String name) { | ||
| protected Optimizer(Graph graph, float learningRate) { |
Craigacp
Sep 19, 2020
Collaborator
Can we have both of these constructors call into the Optimizer(Graph,float,String) one?
Can we have both of these constructors call into the Optimizer(Graph,float,String) one?
JimClarke5
Sep 19, 2020
Author
Contributor
Being that Optimizer is abstract, we really only need one constructor, protected Optimizer(Graph graph, String name, float learningRate) . Of course, we would have to handle a null name, with something like:
this.tf = Ops.create(graph).withName(name == null? getOptimizerName() : name);
Being that Optimizer is abstract, we really only need one constructor, protected Optimizer(Graph graph, String name, float learningRate) . Of course, we would have to handle a null name, with something like:
this.tf = Ops.create(graph).withName(name == null? getOptimizerName() : name);
JimClarke5
Sep 23, 2020
Author
Contributor
CTORS have been changed
CTORS have been changed
| public static String createName(Output<? extends TType> variable, String slotName) { | ||
| return variable.op().name() + "-" + slotName; | ||
| } | ||
|
|
||
| /** |
Craigacp
Sep 19, 2020
Collaborator
Why'd the Javadoc go away?
Why'd the Javadoc go away?
JimClarke5
Sep 19, 2020
Author
Contributor
I am not sure what happened. I had a local copy that I saved and it was there, so will add it back in.
I am not sure what happened. I had a local copy that I saved and it was there, so will add it back in.
JimClarke5
Sep 19, 2020
Author
Contributor
Update pushed
Update pushed
| @@ -305,41 +350,20 @@ private Options() {} | |||
| } | |||
| } | |||
|
|
|||
| /** | |||
Craigacp
Sep 19, 2020
Collaborator
Where'd the javadoc go?
Where'd the javadoc go?
JimClarke5
Sep 19, 2020
Author
Contributor
I have added it back in. Update pushed
I have added it back in. Update pushed
| * @param learningRate the learning rate | ||
| */ | ||
| public final void setLearningRate(float learningRate) { | ||
| if (this.learningRatePlaceholder == null) { |
Craigacp
Sep 19, 2020
Collaborator
Everything seems to have grown a this reference. I don't think that's particularly necessary in these methods, as the argument could be newLearningRate rather than learningRate and then there is no aliasing.
Everything seems to have grown a this reference. I don't think that's particularly necessary in these methods, as the argument could be newLearningRate rather than learningRate and then there is no aliasing.
JimClarke5
Sep 19, 2020
Author
Contributor
I do this out of habit. I can easily change it as you suggest.
I do this out of habit. I can easily change it as you suggest.
JimClarke5
Sep 19, 2020
Author
Contributor
changed setLearningRate to setLearningRate(float newLearningRate), removed spurious this..
Update pushed
changed setLearningRate to setLearningRate(float newLearningRate), removed spurious this..
Update pushed
…ewLearningRate), eliminated spurious "this."
| @@ -280,20 +321,20 @@ protected Op finish(List<Op> updateOperations, String name) { | |||
| /** | |||
| * Sets the learning rate | |||
| * | |||
| * @param learningRate the learning rate | |||
| * @param newLearningRate the new earning rate | |||
Craigacp
Sep 21, 2020
Collaborator
typo - "earning"
typo - "earning"
JimClarke5
Sep 23, 2020
Author
Contributor
OK
OK
...amework/src/main/java/org/tensorflow/framework/optimizers/Optimizer.java
Show resolved
Hide resolved
…raph graph, String name, float learningRate)"", change all the subclass ctors to use this one.
Add Operand<TFloat32> learningRateOperand as an option for learning rate.
|
Just a few snake_case variable names in the tests that need converting to camelCase, and then I'll merge this in. |
| new RMSProp(session.getGraph(), learningRate, decay, momentum, epsilon, centered)) { | ||
| Ops tf = session.getTF(); | ||
| session.setEpsilon(1e-2f); | ||
| float[] var0_init = {1.0F, 2.0F}; |
Craigacp
Sep 30, 2020
Collaborator
Please switch the python style variable names to camelCase.
Please switch the python style variable names to camelCase.
| FloatNdArray mul1 = ND.mul(v, beta); | ||
| FloatNdArray squareG = ND.square(gT); | ||
| FloatNdArray mul2 = ND.mul((1 - beta), squareG); | ||
| return ND.add(mul1, mul2); | ||
| } | ||
|
|
||
| private FloatNdArray calculateParam( | ||
| FloatNdArray param, float lrT, FloatNdArray m, FloatNdArray v, float epsilon) { | ||
| // param - lrT * mT / (np.sqrt(vT) + epsilon) | ||
| FloatNdArray param, float lr_t, FloatNdArray m, FloatNdArray v, float epsilon) { |
Craigacp
Sep 30, 2020
Collaborator
Switch python style name to camelCase.
Switch python style name to camelCase.
This PR requires PR "Initial checkin of Keras Optimzers and helper classes" to be merged first.
Added changeable learning rate to Optimizers. This was done by adding a Placeholder for the learning rate, a Tensor to track the actual learning rate, and adding a Map to map the Placeholder to the Tensor that can be used to "feed" the runner.
Test Sessions were modified to accept a "FeedDict" Map to popullate the feed() of the runner.