in Production

(1)

Analyzing Memory Errors in Production

Markus Weninger

Johannes Kepler University Linz, Austria

Institute for System Software

(2)

WHO ARE WE?

Fixing Memory Problems In Production 2

(3)

WHO ARE WE?

(4)

WHO ARE WE?

(5)

WHO ARE WE?

(6)

WHO ARE WE?

(7)

WHO ARE WE?

(8)

WHO ARE WE?

For what?

To find and fix memory anomalies!

(9)

MOTIVATION

3 Fixing Memory Problems In Production

(10)

MOTIVATION

Jake

(11)

MOTIVATION

Jake

July

(12)

MOTIVATION

Jake

July

(13)

MOTIVATION

Jake

July

(14)

MOTIVATION

Jake

July

(15)

MOTIVATION

Jake

July

(16)

MOTIVATION

Jake

July

(17)

MOTIVATION

Jake

July

(18)

* not actually Jake

(19)

Why is my program crashing?

* not actually Jake

(20)

Why is my program crashing?

* not actually Jake

Let‘s try a

memory

analysis tool!

(21)

PROBLEM

(22)

PROBLEM

(23)

PROBLEM

(24)

PROBLEM

I am new to

all of this!

(25)

PROBLEM

I am new to all of this!

What should

I look for?

(26)

PROBLEM

I am new to all of this!

What should I look for?

When to use

which feature?

(27)

PROBLEM

I am new to all of this!

What should I look for?

When to use which feature?

<another question about memory

analysis> ...

(28)

POLL TIME

(29)

POLL TIME

0 1 2 3 4 5 6

Used a memory analysis tool before

Manually took a heap dump Knows what trace-based memory analysis is

Could describe what a

dominator tree is

Dummy Series 1 Dummy Series 2 Dummy Series 3

(30)

USERS NEED GUIDANCE

(31)

MEMORY ANALYSIS 101

(32)

MEMORY ANALYSIS 101

(33)

MEMORY ANALYSIS 101

Top-down analysis

(34)

MEMORY ANALYSIS 101

Top-down analysis

GC root

(35)

MEMORY ANALYSIS 101

Top-down analysis

GC root

(36)

MEMORY ANALYSIS 101

Top-down analysis

GC root

... ... ... ...

(37)

MEMORY ANALYSIS 101

Top-down analysis

GC root

... ... ... ...

...

(38)

MEMORY ANALYSIS 101

Top-down analysis

GC root

... ... ... ...

...

(39)

MEMORY ANALYSIS 101

Top-down analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

...

(40)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

...

(41)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

... ...

(42)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

... ...

(43)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

... ...

(44)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

... ...

(45)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

... ...

(46)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

GC root GC root

... ...

(47)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

GC root GC root

1 Mio. 10,000

10,000 10,000

1 Mio.

... ...

(48)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

GC root GC root

1 Mio. 10,000

10,000 10,000

1 Mio.

... ...

(49)

MEMORY ANALYSIS 101

Top-down analysis Bottom-up analysis

GC root

... ... ... ...

Which single object / GC root keeps the most other objects alive?

GC root GC root

1 Mio. 10,000

10,000 10,000

1 Mio.

...

Analyze why a certain group of objects accumulates over time

...

(50)

DOMINATOR TREE

GC Roots

L1 L2

N1 N2 N3 N4 N5

D1 D2 D3 D4 D5

D6 D7 D8

N6 N7 N8 N9

N10

Object Reference Graph

(51)

DOMINATOR TREE

GC Roots

L1 L2

N1 N2 N3 N4 N5

D1 D2 D3 D4 D5

D6 D7 D8

N6 N7 N8 N9

N10

Dom. Tree Root

L1 L2

N1 N2 N3 N4 N5 D1

D2 D3 D4 D5

D6 D7

D8

N6 N7 N8 N9

N10

Object Reference Graph Dominator Tree

(52)

DOMINATOR TREE

GC Roots

L1 L2

N1 N2 N3 N4 N5

D1 D2 D3 D4 D5

D6 D7 D8

N6 N7 N8 N9

N10

Dom. Tree Root

L1 L2

N1 N2 N3 N4 N5 D1

D2 D3 D4 D5

D6 D7

D8

N6 N7 N8 N9

N10

Object Reference Graph Dominator Tree

(53)

DOMINATOR TREE

GC Roots

L1 L2

N1 N2 N3 N4 N5

D1 D2 D3 D4 D5

D6 D7 D8

N6 N7 N8 N9

N10

Dom. Tree Root

L1 L2

N1 N2 N3 N4 N5 D1

D2 D3 D4 D5

D6 D7

D8

N6 N7 N8 N9

N10

Object Reference Graph Dominator Tree

What happens if we could free L1?

(54)

DOMINATOR TREE

GC Roots

L1 L2

N1 N2 N3 N4 N5

D1 D2 D3 D4 D5

D6 D7 D8

N6 N7 N8 N9

N10

Dom. Tree Root

L1 L2

N1 N2 N3 N4 N5 D1

D2 D3 D4 D5

D6 D7

D8

N6 N7 N8 N9

N10

Object Reference Graph Dominator Tree

These can be

collected too

(55)

(56)

Thanks Markus, but that

seems like a lot to learn...

(57)

Thanks Markus, but that

seems like a lot to learn...

(58)

Thanks Markus, but that seems like a lot to learn...

Okay, let us try to

improve that in AntTracks

(59)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

(60)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X()

(61)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X()

(62)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X()

Trace

file

(63)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y()

Trace

file

(64)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z()

Trace

file

(65)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

Trace

file

(66)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move

Trace

file

(67)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move

Trace

file

(68)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

Trace

file

(69)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y()

Trace

file

(70)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace

file

(71)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

(72)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1

(73)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1

(74)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1 Heap 2

(75)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1 Heap 2 Heap n

…

(76)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1 Heap 2 Heap n

…

(77)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1 Heap 2 Heap n

…

(78)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1 Heap 2 Heap n

…

(79)

ANTTRACKS SYSTEM OVERVIEW

AntTracks VM Application t

M ut at or phase M ut . phase GC pha s e

new X() new Y() new Z() new Y()

move move move

new Y() new Z()

Trace file

AntTracks Analyzer Memory anomaly detection

Memory leak analysis

Heap 1 Heap 2 Heap n

…

(80)

GUIDED EXPLORATION

(81)

GUIDED EXPLORATION

Detection describes the task of automatically detecting a potential problem, i.e.,

a suspicious pattern.

(82)

GUIDED EXPLORATION

Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern.

Highlighting the relevant region on the UI helps users to understand where the

automatically gained insight can be found if the view was inspected manually.

(83)

GUIDED EXPLORATION

Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern.

Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually.

Explanation of the highlighted visualization helps users to interpret it and

explains concepts that are needed for this interpretation.

(84)

GUIDED EXPLORATION

Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern.

Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually.

Explanation of the highlighted visualization helps users to interpret it and explains concepts that are needed for this interpretation.

Suggestions on which steps could / should be taken next make it easier for the

user to understand what operations are possible and why they might be useful.

(85)

GUIDED EXPLORATION

Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern.

Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually.

Explanation of the highlighted visualization helps users to interpret it and explains concepts that are needed for this interpretation.

Suggestions on which steps could / should be taken next make it easier for the

user to understand what operations are possible and why they might be useful.

(86)

DEMO: EASYTRAVEL

(87)

DEMO: OVERVIEW

(88)

DEMO: OVERVIEW

(1) DETECTION + (2) HIGHLIGHTING

(89)

DEMO: OVERVIEW

(3) EXPLANATION + (4) SUGGESTION

(90)

DEMO: EVOLUTION

(91)

DEMO: EVOLUTION

(92)

DEMO: EVOLUTION

Date objects allocated in the constructor of Base are the major suspects for a memory leak since about 30% of the overall heap growth is accounted to them.

Inspect who keeps them alive.

(93)

DEMO: GRAPH VIEW

(94)

DEMO: GRAPH VIEW

(95)

DEMO: GRAPH VIEW

(96)

DEMO: GRAPH VIEW

(97)

DEMO: GRAPH VIEW

Too many ArrayList<Location>

are added here.

(98)

ANOTHER DEMO: FINAGLE-HTTP

(99)

ANOTHER DEMO: FINAGLE-HTTP

(100)

ANOTHER DEMO: FINAGLE-HTTP

(101)

ANOTHER DEMO: FINAGLE-HTTP

(102)

ANOTHER DEMO: FINAGLE-HTTP

(103)

ANOTHER DEMO: FINAGLE-HTTP

(104)

ANOTHER DEMO: FINAGLE-HTTP

(105)

ANOTHER DEMO: FINAGLE-HTTP

(106)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request)

(107)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

(108)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

(109)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

val h = { rep: http.Response =>

totalLength += rep.content.length }

for (i <- 0 until NUM_REQUESTS {

Await.result(response.onSuccess(h)) }

(110)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

val h = { rep: http.Response =>

totalLength += rep.content.length }

for (i <- 0 until NUM_REQUESTS {

Await.result(response.onSuccess(h)) }

(111)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

val h = { rep: http.Response =>

totalLength += rep.content.length }

for (i <- 0 until NUM_REQUESTS {

Await.result(response.onSuccess(h)) }

(112)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

val h = { rep: http.Response =>

totalLength += rep.content.length }

for (i <- 0 until NUM_REQUESTS {

Await.result(response.onSuccess(h)) }

(113)

DEMO: FINAGLE-HTTP

val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) {

Await.result(response.onSuccess { rep: http.Response =>

totalLength += rep.content.length })

}

val h = { rep: http.Response =>

totalLength += rep.content.length }

for (i <- 0 until NUM_REQUESTS {

Await.result(response.onSuccess(h)) }

(114)

TAKE-AWAYS

(115)

TAKE-AWAYS

Memory Analysis

Top-down Bottom-up Heap dumps

Trace-based

... and 100 other things

(116)

TAKE-AWAYS

Memory Analysis Problem

Top-down Bottom-up Heap dumps

Trace-based

... and 100 other things

New users need guidance to get

started faster It is not enough to

„throw tools at them“

(117)

TAKE-AWAYS

Memory Analysis Problem Guided Exploration

Top-down Bottom-up Heap dumps

Trace-based

... and 100 other things

New users need guidance to get

started faster It is not enough to

„throw tools at them“

Detection Highlighting

Explanation

Suggestion

(118)

TAKE-AWAYS

Memory Analysis Problem Guided Exploration

Top-down Bottom-up Heap dumps

Trace-based

... and 100 other things

New users need guidance to get

started faster It is not enough to

„throw tools at them“

Detection Highlighting

Explanation Suggestion

Markus Weninger

Johannes Kepler University

Linz, Austria

markus.weninger@jku.at

http://mevss.jku.at/AntTracks

http://bit.ly/weninger_ssw

(119)

Additional Notes

 Some icons made by Freepik & Smashicons from https://www.flaticon.com

 Some photos made by bruce mars and Isaque Pereira from Pexels